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DISTRIBUTED  LEARNING  GUIDELINES 


INTRODUCTION 

Delivery  Order  (DO)  5  of  the  Digital  Skills  Training  Research  (DSTR)  project  is  aimed  at 
the  identification  of  digital  skill  acquisition  and  retention  factors.  Its  overall  purpose  is  to 
document  patterns  of  knowledge/skill  loss  for  Military  Intelligence  (MI)  and  Field  Artillery  (FA) 
and  to  introduce  prototype  training  interventions  to  improve  skill  retention.  These  interventions 
are  based  on  constructivist  learning  approaches  that  require  the  learner  to  relate  current  content 
to  previously  learned  material  and  to  actively  build  on  prior  knowledge/skills. 

An  additional  task  under  DO  5  concerns  the  development  of  guidelines  for  distributed 
learning  (DL),  an  increasingly  viable  means  for  delivering  instruction  in  the  Army.  Distributed 
learning,  alternatively  termed  distance  learning,  has  been  defined  by  the  Los  Alamos  National 
Laboratory  as  “structured  learning  that  takes  place  without  the  physical  presence  of  the 
instructor.”  The  Air  Education  and  Training  Command  defines  DL  as  “any  method  of  presenting 
training  that  is  interactive  and  in  which  students  are  physically  separate  from  the  instructor.” 
Hedberg,  Brown,  and  Arrighi  (1997)  define  DL  by  the  two  dimensions  that  comprise  its 
delivery — space  and  time.  The  following  DL  combinations  are  possible: 

1.  Same-place  and  same-time:  synchronous  delivery  with  instructor  and  learner 
collocated. 

2.  Different-place  and  same-time:  synchronous  delivery  with  instructor  and  learner 
remotely  located. 

3.  Same-place  and  different-time:  asynchronous  delivery  with  instructor  and  learner 
collocated. 

4.  Different-place  and  different-time:  asynchronous  delivery  with  instructor  and  learner 
remotely  located. 

DL  enables  the  instructor  to  be  virtual  when  learning  is  facilitated  via  technology 
(computers,  networks,  databases).  Instructional  content  must  be  designed  to  enhance  learning 
with  or  without  the  physical  presence  of  an  instructor.  It  is  this  instructional  design  issue  that  is 
at  the  heart  of  the  controversy  regarding  DL.  The  following  questions  should  be  addressed 
regardless  of  whether  the  content  is  delivered  at  a  distance,  or  more  conventionally  in  a 
classroom  setting. 
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•  What  are  we  trying  to  teach  (skills,  knowledge;  facts,  rules,  principles,  abstract 
concepts)? 

•  What  is  the  level  of  preparedness  of  learners? 

•  What  is  their  learning  background  relative  to  the  content? 

•  How  will  we  know  when  learning  has  occurred?  How  can  we  document  learning? 

•  How  much  learning  must  occur  to  achieve  mastery  of  the  material? 

•  What  are  consequences  of  not  learning  the  content? 

•  How  do  we  measure  transfer  from  the  learning  environment  to  the  applied  setting? 

•  How  frequently  must  learners  practice  or  be  retrained  to  sustain  proficiency? 

While  DL  is  not  new — correspondence  courses  are  almost  as  old  as  the  postal  system — 

there  has  been  a  dramatic  increase  in  its  use  as  computer  and  Internet  technology  has  evolved. 
Initially,  DL  growth  was  associated  with  satellite-based  learning  systems  (one-way  and  two-way 
audio  and  video).  For  the  past  decade,  DL  also  has  been  delivered  via  the  Web  and  on 
organizational  intranets.  The  International  Data  Corporation  estimates  that  IT-related  Web-based 
training  will  surge  to  more  than  $6  billion  by  2002 — a  compounded  annual  growth  rate  of  nearly 
95  percent  over  1999  expenditures.  Ragan’s  Strategic  Training  Report  (1999)  predicts  even  more 
aggressive  growth  rates,  estimating  that  corporate  Web-based  training  expenditures  by  American 
corporations  will  be  in  the  $10  billion  range  by  2002.  Spurred  by  DL  initiatives  among  all 
military  services,  and  especially  the  Army,  Department  of  Defense  (DoD)  DL  budgets  are 
increasing  exponentially.  Currently,  the  FY  2001  DoD  budget  has  approximately  $7  billion 
earmarked  for  training  and  recruiting  (DoD,  2000). 

Numerous  factors  are  affecting  DL’s  projected  growth  rates.  Distance  learning  is 
accessible,  increasingly  interactive  and  engaging,  allows  for  remote  collaboration  among 
students  and  instructors,  and  significantly  reduces  travel  costs.  For  these  and  other  reasons, 
military  DL  has  been  reported  to  have  a  very  encouraging  future  (Barry  and  Runyan,  1995; 
Chute,  Thompson,  and  Hancock,  1999). 

But  DL  is  not  a  panacea.  Indiscriminate  use  of  the  technology  for  its  own  sake  has  little 
or  no  learning  benefits.  Like  conventional  training  and  education,  poorly  designed  on-line 
content  is  almost  certain  to  alienate  learners.  Clark  (1983)  has  asserted  that  media  do  not 
influence  learning  under  any  conditions.  Instead,  evidence  suggests  that  “media  are  mere 
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vehicles  that  deliver  instruction  but  do  not  influence  student  achievement  any  more  than  the 
truck  that  delivers  our  groceries  causes  changes  in  our  nutrition.” 

In  a  benchmark  report,  Russell  (1999)  cited  some  350  studies  reporting  “no  significant 
difference”  between  technology-delivered  DL  and  conventional  classroom  instruction.  Phipps 
and  Merisotis  (1999),  Machtmes  and  Asher  (2000),  Champagne  and  Wisher  (2000),  Wisher, 
Champagne,  Pawluk,  Eaton,  and  Cumow  (1999),  and  others  have  asserted  that  Russell’s  findings 
are  grounded  on  ill-conceived  and  poorly  designed  studies  that  are  characterized  by  numerous 
shortfalls  including: 

•  Lack  of  controls 

•  No/ineffective  experimental  design 

•  Confounding  of  student  experience  with  learning 

•  Non-random  assignment  of  participants 

•  Subjective  assessments  of  effectiveness 

•  Lack  of  explicitly  defined  mastery  criteria 

•  Absence  of  pre-tests  and  in  many  cases,  post-tests 

•  No  longitudinal  follow-ups 

Wisher  (2000)  recently  described  four  principal  threats  (Table  1)  to  the  internal  validity 
of  DL  research.  These  threats  confound  the  interpretation  of  DL  effects  on  learning  outcomes. 


Table  1. 

Threats  to  Internal  Validity  of  DL  Findings 


Threat 

Description 

History 

Skill/knowledge  differences  may  be  due  to  learning  or 
experience  not  attributable  to  DL. 

Maturation 

Skill/knowledge  differences  may  be  due  to  learners  becoming 
more  fatigued  or  less  interested  in  the  program  over  time. 

Mortality 

Learners  may  become  discouraged  and  drop  out  of  DL 
program  so  that  average  post-test  scores  exceed  average  pre¬ 
test  scores. 

Test  Sensitization 

Learners  may  be  cued  by  pre-test  measures  to  focus  on  those 
knowledge-based  items  that  enhance  their  post-test  scores 
regardless  of  program  content. 
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Jones  and  Paolucci  (1996)  have  estimated  that  less  than  5  percent  of  published  DL 
research  is  sufficiently  empirical,  quantitative,  and  valid  to  support  conclusions  about  DL 
learning  outcomes.  Research  that  can  be  meaningfully  used  to  predict  DL  success  is  woefully 
inadequate  (Navarro  and  Shoemaker,  2000).  The  levels  of  technology  effectiveness  on  enhancing 
the  learning  process  remain  to  be  seen  (Recker,  1997).  Bonk  and  Wisher  (2000)  have  outlined 
the  requirements  for  collaborative  learning  environments  that  are  learner-centered  and  can  be 
applied  within  networked  simulations.  They  further  describe  differences  between  education  and 
training  that  must  be  considered  in  formulating  our  instructional  design  and  delivery  strategies. 

Guidelines  are  needed  for  selection  and  application  of  DL  methods  and  media  for  applied 
researchers,  developers,  and  practitioners.  These  guidelines  should  be  incorporated  with 
instructional  design  and  content  issues,  experimental  design,  and  measurement  and  evaluation 
factors.  DL  guidelines  should  stem  from  assessments  of  media  and  methods  relative  to  types  of 
knowledge  and  skills  to  be  imparted.  Rapid  developments  in  training  delivery  systems  demand 
that  research  continuously  re-examine  which  technology  is  best  suited  for  particular  skill  and 
knowledge  types  (Machtmes  and  Asher,  2000). 

The  majority  of  studies  aimed  at  DL  effectiveness  are  based  on  educational  (vs.  training) 
settings  and  on  the  collection  of  observational  data  (Champagne,  Wisher,  Pawluk,  and  Cumow, 
1999).  Very  little  research  has  been  conducted  to  determine  the  interrelationships  among 
individual  learning  styles  and  DL  content  (Liu  and  Ginther,  1999).  The  present  research  was 
aimed  at  developing  DL  guidelines  stemming  from  empirical  studies  that  contain  measurable  and 
controllable  training  applications. 


PURPOSE  AND  SCOPE 

This  task  was  designed  to  provide  a  framework  for  interpreting  the  effectiveness  of  DL 
methods/media  for  facilitating  various  types  of  learning.  One  of  the  most  influential  models  of 
training  evaluation  over  the  past  30  years  is  Kirkpatrick’s  four-level  model  (Kirkpatrick,  1998). 
He  suggests  that  training  evaluations  should  address  four  general  areas:  reaction,  learning, 
behavior,  and  results.  The  model  is  presented  in  Table  2  (adapted  from  Childs,  1996). 
Kirkpatrick’s  model  highlights  the  various  levels  that  should  be  addressed  as  part  of  training 
system  evaluations. 
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Table  2. 


Kirkpatrick’ s  Four  Levels  of  Evaluation 


Level  and  Type 

What  is  Measured  and  Evaluated 

Measurement  Method 

1:  Learner  Reaction 

•  Satisfaction 

•  (The  “Smile  Factor”) 

•  Course  Materials  Ratings 

•  Content  Delivery  Effectiveness 

•  End  of  Training 
Evaluation  or  Critique 

2:  Learning 

•  New  Knowledge,  Skills,  and  Attitudes 
(KSA)  Attainment 

•  Objectives  Mastery 

•  Final  Examination 

•  Performance  Exercise 

•  Pre-/Post-Tests 

3:  Application 

•  Use  of  KSA  on  Job 

•  Training  Transfer 

•  Individual  or  Team  Improvement 

•  Job  Performance 
Outcomes 

4:  Results 

•  Return  on  Training  Investment 

•  Organizational  or  Corporate  Benefits 

•  Cost  Benefits  Analysis 

•  Business  Outcomes 

The  focus  of  the  present  research  was  on  learning  outcomes  (Level  2  evaluations) 
generated  from  empirical  studies  of  training  effectiveness  (use  of  treatment  and  control  groups 
with  at  least  ordinal-level  measures  and  inferential  statistics).  Generally,  research  has  shown  that 
learner  attitudes  are  not  effective  predictors  of  learning  outcomes  (Moore  and  Kearsley,  1996). 
Yet  as  Figure  1  shows,  the  American  Society  of  Training  and  Development  (ASTD)  indicates 
that  the  vast  majority  of  corporate  training  assessments  focus  on  how  learners  perceive  their 
learning  experience  (Kirkpatrick  Level  1)  rather  than  on  how  or  what  they  learn  (ASTD,  1997). 
Fewer  than  one-half  of  the  100  large  companies  surveyed  in  1997  indicated  that  they  assess 
learning  outcomes  (Level  2)  and  only  about  10%  reported  training  transfer  evaluations  (Level  3). 
We  are  not  aware  of  counterpart  data  gathered  on  the  DoD,  but  would  expect  the  incidence  of 
Level  2  and  3  evaluations  to  be  higher  for  military  settings.  Based  on  trends  across  the  three 
years  reported  by  ASTD,  there  is  little  reason  to  expect  significant  changes  in  the  Figure  1  rates 
since  1997.  Although  corporations  are  increasingly  adopting  technology-based  training,  there  is 
little  emphasis  on  evaluating  training  effectiveness  via  learning  outcomes. 
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Figure  1.  Levels  of  Training  Assessments  as  Reported  by  American  Business 

DL  data  not  based  on  empirical  studies  were  considered  out  of  scope  for  this  effort.  Non- 
empirical  DL  studies  and  references  are  summarized  in  Appendix  A. 

Our  research  focused  on  studies  of  training  effectiveness  conducted  since  1996.  We  also 
reviewed  earlier  studies  provided  by  the  Army  Research  Institute  (ARI)  for  the  purpose  of 
potential  inclusion  in  the  evaluation.  A  matrix  was  generated  to  evaluate  DL  guidelines.  Our  goal 
was  to  include  in  the  matrix  knowledge/skill  types  addressed  by  empirical  investigations  and  DL 
methods/media  utilized  to  deliver  the  training.  We  were  less  concerned  with  educational 
applications  (university  studies)  than  with  those  aimed  at  performance-based  training.  Similarly, 
greater  emphasis  was  placed  on  applied  training  (industrial  and  military  applications)  than  on 
academic  learning  environments.  Kindergarten- 12  studies  were  excluded  since  the  goal  was  to 
focus  on  adult  learning.  Priority  was  given  to  those  studies  that  assessed  the  effectiveness  of  DL 
methods  relative  to  traditional  classroom-based  instruction.  Studies  that  assessed  measurable 
learning  outcomes  to  include  statistical  assessments  of  learning  were  considered  higher  priority 
than  those  that  were  descriptive  in  nature. 
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METHOD 


To  accomplish  the  objectives  of  the  study,  we  followed  the  general  approach  described 

below. 

1.  Identification  and  review  of  approximately  200  DL  studies  and  other  data  sources 
from  ARI  and  from  literature  and  Web  site  reviews. 

TRW  employed  several  sources  to  gather  the  study  data.  These  included  the  professional 
DL  literature,  ARI  DL-related  documents,  and  DL  Web  sites.  We  collected  some  200  DL 
references  published  in  professional  journals  such  as  the  American  Journal  of  Distance 
Education,  Distance  Education,  and  the  Journal  of  Asynchronous  Learning  Networks.  We  also 
reviewed  DL  conference  proceedings  such  as  those  published  by  the  University  of  Wisconsin  as 
well  as  government  and  industry  Web  sites  containing  pertinent  information.  Research  was 
reviewed  from  ARI  and  other  DoD  and  non-military  government  organizations  concerned  with 
DL.  We  conducted  DL  literature  reviews  in  the  University  of  New  Mexico  library  and  purchased 
proceedings  from  DL  industry  conferences  and  professional  meetings. 

2.  Discussions  with  ARI  and  other  DL  experts. 

We  conferred  with  ARI  and  DL  experts  from  the  University  of  New  Mexico  and  Indiana 
University.  We  also  attended  professional  conferences  (e.g.,  the  E-leaming  conference)  that 
included  DL-related  presentations. 

3.  Selection  and  rating  of  DL  studies. 

From  the  initial  study  population,  we  applied  selection  criteria  (post-1996  publication 
dates,  empirical,  training-oriented,  focused  on  learning  outcomes)  to  arrive  at  those  DL  studies  to 
be  rated  within  the  evaluation  matrix.  We  evaluated  those  studies  against  12  evaluation  factors 
gleaned  from  the  literature.  We  constructed  an  evaluation  matrix  and  rating  tool,  and  assigned 
the  studies  to  the  matrix  based  on  DL  media  (video,  audio,  CBT)  and  knowledge  types 
(perceptual-motor,  procedural,  cognitive).  We  then  rated  the  effectiveness  of  the  DL  media  for 
training  various  skill/knowledge  types  and  computed  effect  sizes  where  possible.  Under  Results, 
we  provide  a  brief  description  of  our  interpretation  of  findings  and  discuss  the  relevance  of 
matrix  summary  data. 


7 


RESULTS 


Consistent  with  findings  reported  by  Wisher  et  al.  (1999),  a  significant  majority  of  the 
200  studies  we  reviewed  contain  anecdotal  and/or  opinion-based  data.  Most  of  the  studies 
reviewed  lack  valid  measurements  of  learning  outcomes  (Kirkpatrick  Level  2  measures)  and  the 
majority  include  no  (or  flawed)  experimental  designs.  While  many  of  the  studies  contain  useful 
information  for  designing,  developing,  delivering,  and  managing  DL  content,  data  were  unusable 
as  reported.  Of  the  study  population  reviewed,  only  15  DL  studies  met  selection  criteria 
described  under  Purpose  and  Scope.  The  studies  are  summarized  in  Table  3.  The  purpose  of 
downselecting  to  these  15  studies  was  to  rate  DL  method/media  effectiveness  for  training  various 
skill/knowledge  types. 


Table  3. 

DL  Studies  Meeting  Selection  Criteria 


Ref 

No. 

Authors 

Title 

Description 

1 

Bramble  & 
Martin  (1995) 

The  Florida  Teletraining 

Project:  Military  Training  via 
Two-way  Compressed  Video 

For  five  different  content  areas, 
students  trained  via  the  Army’s 
TNET  system  performed 
significantly  better  (p<0.001)  on 
post-tests  than  pre-tests.  Repeated 
measures  design  with  no  control 
group. 

2 

Drenth, 
Kubisiak,  & 
Borman 
(2000) 

An  Examination  of  the 
Effectiveness  of  Distance 
Learning  for  the  Battle  Staff 

NCO  Course 

VTT  groups  performed  almost 
identically  on  Job  Knowledge  Tests 
(p=0.57)  to  groups  trained  in 
residence  at  Fts.  Bliss  and  McCoy. 

3 

Howard, 

Henry, 

Kinney,  & 
Dannhaus 
(1991) 

Distributed  Training  Strategy 
Training  Effectiveness 

Analysis:  MOS  63W  Desktop 
Video  Pilot  Study 

Groups  trained  via  desktop  video 
performed  significantly  better 
(p<0.01)  than  controls  on  4  of  6 
lessons  and  equally  as  good  on  the 
other  two. 

4 

Keene  &  Cary 
(1990) 

Effectiveness  of  Distance 
Education  Approach  to  U.S. 
Army  Reserve  Component 
Training 

Students  receiving  VTT  achieved 
significantly  higher  scores 
(p<0.001)  than  controls  on  3  of  4 
learning  outcome  measures. 

8 


Ref 

No. 

Authors 

Title 

Description 

5 

Moshinskie, 
Jarvis,  Hobbs, 

&  Roden 
(1996) 

The  Effects  of  Using  Distance 
Learning  Technologies  in 
Training  Rural  EMS  Providers 
When  Learner-Centered 

Designs  are  Used 

EMS  students  trained  on  advanced 
life  support  courses  via  2-way  audio 
and  2-way  audio/video  (satellite- 
based)  performed  as  well  as  those 
trained  in  classroom. 

6 

Phelps,  Wells, 
Ashworth,  & 
Hahn  (1991) 

Effectiveness  and  Costs  of 
Distance  Education  Using 
Computer-Mediated 
Communication 

CMC  group  performed  significantly 
better  (p<0.001)  than  controls  on 
leadership  test  outcomes.  For 
engineering  course,  no  significant 
differences  in  test  scores  were 
found. 

7 

Simpson, 

Pugh,  & 

Parchman 

(1991) 

Empirical  Comparison  of 
Alternative  Video  Teletraining 
Technologies 

Two-way  video  does  not  facilitate 
student  learning  relative  to  the  use 
of  one-way  video  during  VTT  to 
support  Admin  and  Operations 
courses.  No  control  group. 

8 

Simpson, 

Pugh,  & 

Parchman 

(1992) 

The  Use  of  Videoteletraining  to 
Deliver  Hands-on  Training: 
Concept  Test  and  Evaluation 

VTT  groups  trained  remotely 
performed  as  well  as  those  trained 
in  local  classrooms  on  three  end-of- 
course  hands-on  skills  tests 
(p>0.05). 

9 

Simpson, 
Wetzel,  & 

Pugh  (1995) 

Delivery  of  Division  Officer 
Navy  Leadership  Training  by 
Videoteletraining:  Initial 

Concept  Test  and  Evaluation 

Navy  Leadership  tests  showed  that 
groups  trained  via  VTT  local  and 
VTT  remote  performed  nearly 
identically  to  those  trained  in  the 
classroom. 

10 

Wetzel  (1996) 

Distributed  Training 

Technology  Project:  Final 

Report 

Four  courses  delivered  via  VTT. 
Statistics  are  difficult  to  interpret. 
Remote  VTT  groups  performed  as 
well  as  local  VTT  groups  on  most 
measures. 

11 

Wetzel,  Pugh, 
Van  Matre,  & 
Parchman 
(1996) 

Videoteletraining  Delivery  of  a 
Quality  Assurance  Course  with 
a  Computer  Laboratory 

Same  as  Wetzel  (1996),  with  results 
reported  only  for  one  (QA)  course. 
No  significant  difference  between 
local  and  remote  VTT  groups  on 
exam  grades. 

9 


Ref 

No. 

Authors 

Title 

Description 

12 

Wetzel, 

Radtke, 
Parchman,  & 
Seymour 
(1996) 

Delivery  of  a  Fiber  Optic  Cable 
Repair  Course  by 
Videoteletraining 

Procedural  errors  were  no  higher 
for  VTT  group  than  for  controls. 
Learning  outcomes  were  not 
statistically  different  among  VTT 
local,  VTT  remote,  and  control 
groups. 

13 

Whetzel, 

Felker,  & 
Williams 
(1996) 

A  Real  World  Comparison  of 
the  Effectiveness  of  Satellite 
Training  and  Classroom 

Training 

U.S.  Postal  Service  employees 
trained  via  satellite  (VTT) 
performed  significantly  better  than 
classroom-trained  group 

14 

Wisher  & 

Cumow 

(1999) 

Perceptions  and  Effects  of 

Image  Transmissions  during 
Internet-Based  Training 

Video  capability  for  Intemet-based 
audiographics  course  on 
information  operations  produced  no 
better  learning  than  without  video. 

15 

Wisher,  Priest, 
&  Glover 
(1997) 

Audio  Teletraining  for  Unit 
Clerks:  A  Cost-Effectiveness 
Analysis 

Audio  teletrained  group  performed 
significantly  better  (higher  Go 
rates;  p<0.001)  than  control  group. 

Prior  to  rating  the  effectiveness  of  each  DL  method/medium  for  training  various 
skill/knowledge  types,  we  assessed  the  DL  studies  against  12  evaluation  factors.  This  assessment 
is  summarized  in  Figure  2.  While  these  evaluation  factors  are  not  intended  to  be  comprehensive, 
they  represent  desirable  characteristics  of  DL  studies  as  gleaned  from  our  literature  review  and 
discussed  with  ARI.  Figure  2  includes  the  studies  identified  for  evaluating  the  effectiveness  of 
methods/media  for  training  the  various  DL  skill  and  knowledge  types.  If  studies  contained 
sufficient  information  to  verify  that  the  evaluation  factor  was  addressed,  a  check  mark  was 
assigned  to  the  applicable  cell.  No  attempt  was  made  to  rate  the  degree  of  a  factor’s 
effectiveness,  only  that  it  was,  or  was  not,  addressed  by  the  study.  Neither  was  an  attempt  made 
to  differentially  weight  factors  since  any  factor  could  be  more  or  less  critical  depending  on  study 
context  and  objectives. 

While  the  majority  of  the  Figure  2  studies  included  sound  experimental  and  instructional 
designs  and  various  statistical  analyses,  only  37%  of  the  cells  include  check  marks. 

With  regard  to  study  compliance  with  the  individual  evaluation  factors  in  Figure  2: 

•  Three  factors — instructional  design,  experimental  design,  and  statistical  analyses — 
were  addressed  by  virtually  all  of  the  studies. 
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•  No  study  addressed  more  than  six  evaluation  factors. 

•  Only  two  studies  are  non-military  in  nature;  these  assessed  Emergency  Medical 
Service  personnel  (Study  5)  and  U.S.  Postal  Service  workers  (Study  13). 

•  No  study  reported  randomly  assigning  students  to  treatment  and  control  groups. 

•  Only  three  studies  reported  the  attrition  rates  of  DL  students. 

•  Specific  roles  and  responsibilities  for  students  in  the  learning  process  were  not 
covered  (three  studies  reported  preparation  and  planning  efforts  on  the  part  of 
students). 

•  Over  half  of  the  reports  alluded  to  instructor  initiative  in  assisting/mentoring  students. 

•  Only  three  studies  included  the  use  of  multiple  DL  technologies. 

•  None  of  the  studies  reported  the  use  of  digital  libraries. 

•  Only  two  studies  incorporated  training  transfer  evaluations  (Kirkpatrick  Level  3). 

•  Two  studies  included  program/course  effectiveness  evaluations. 


Evaluation 

Factors 


A''  ,A.V  ~<S>  .xCv 


^  ^  //  y  y  y  y  y  y 


1.  Bramble  and  Martin,  1995 

✓ 

y* 

✓ 

y 

2.  Drenth,  Kubisiak,&  Borman,  2000 

✓ 

✓ 

y 

y 

y 

3.  Howard,  Henry,  Kinney,  &  Dannhaus,  1991 

✓ 

✓ 

y 

4.  Keene  and  Cary,  1990 

✓ 

✓ 

y 

5.  Moshinskie,  Jarvis,  Hobbs,  and  Roden,  1996 

✓ 

✓ 

y 

y 

y 

6.  Phelps,  Wells,  Ashworth,  &  Hahn,  1991 

✓ 

✓ 

y 

y 

y 

y 

7.  Simpson,  Pugh,  and  Parchman,  1991 

✓ 

y* 

y 

y 

y 

y 

8.  Simpson,  Pugh,  and  Parchman,  1992 

✓ 

✓ 

y 

y 

y 

y 

9.  Simpson,  Wetzel,  and  Pugh,  1995 

✓ 

y 

y 

y 

10.  Wetzel  (1996) 

✓ 

✓ 

y 

y 

y 

11.  Wetzel,  Pugh,  Van  Matre,  &  Parchman  (1996) 

✓ 

y 

y 

y 

12.  Wetzel,  Radtke,  Parchman,  &  Seymour,  1996 

✓ 

y 

y 

13.  Whetzel,  Felker,  and  Williams  (1996) 

y 

y 

14.  Wisher  and  Curnow  (1999) 

V 

✓ 

y 

y 

y 

15.  Wisher,  Priest,  and  Glover  (1997) 

y 

y 

y 

y 

*  No  control  group. 

1  Military  training  schedules  often  preclude  random  assignment  of  learners  to  experimental  and  control  conditions. 

2  Kirkpatrick  Level  1  measures  were  not  included  since  they  are  not  empirical. 

3  Due  to  standardization  requirements,  individual  learning  styles  are  seldom  addressed  in  military  studies. 

4  For  military  studies,  learner  attrition  is  generally  low. 

Figure  2.  DL  Evaluation  Factors  Applied  to  the  Selected  Studies 
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Figure  2  suggests  that  DL  evaluations  are  truly  multidimensional  and  that  DL  research 
conducted  over  the  past  five  years  is  addressing  only  a  few  of  those  dimensions.  Of  the  200 
studies  initially  reviewed,  those  represented  in  Figure  2  are  the  most  empirically  rigorous.  Thus, 
the  DL  literature  continues  to  be  dominated  by  anecdotal  or  other  qualitative  information.  Even 
data  on  the  incidence,  design,  and  cost  of  DL  for  educational  and  training  purposes  are  suspect 
due  to  the  absence  of  standards  for  defining  their  use.  The  Advanced  Distributed  Learning 
(ADL)  initiative,  discussed  under  Current  Programs,  promises  to  incorporate  standards  and 
guidelines  for  improving  the  validity  and  reliability  training  effectiveness  data. 

We  used  the  taxonomy  included  in  Appendix  B  to  classify  skill/knowledge  types.  This 
classification  is  largely  derived  from  Bloom’s  (1956)  Taxonomy  (Table  4)  for  characterizing 
types  of  learning.  The  rating  of  the  effectiveness  of  the  method/medium  for  imparting  that 
particular  skill/knowledge  type  was  then  determined  for  that  cell  along  with  an  effect  size,  if 
reported. 


Table  4. 

Bloom ’s  Taxonomy — The  Six  Levels  of  Learning 


Learning  Style 

Definition 

Knowledge 

Universal  axioms  and  principles:  terminology,  facts,  specifics, 
conventions,  and  criteria.  The  ability  to  remember  (recall)  previously 
learned  materials.  Arranges,  defines,  describes,  duplicates,  identifies, 
knows,  labels,  lists,  matches,  memorizes,  names,  outlines,  recalls, 
recognizes,  repeats,  reproduces,  selects,  states. 

Comprehension 

Use  of  material  in  a  form  different  from  the  way  it  was  learned,  i.e., 
translation,  interpretation,  extrapolation. 

Application 

Problem  solving  and  the  ability  to  do  case  study. 

Analysis 

Elements  and  relations  of  organizational  principles:  an  ability  to  break 
down  a  situation  into  its  parts. 

Synthesis 

Production  of  a  unique  communication  which  puts  a  field  in  a  broader 
context,  production  of  a  plan  or  set  of  objectives. 

Evaluation 

Good  or  bad,  workable,  helpful,  etc.?  What  values  does  it  uphold? 
Internal  vs.  external  evidence. 

Our  learning  effectiveness  rating  procedure  was  designed  to  minimize  subjectivity.  The 
procedure  used  a  scale  (Figure  3)  with  clearly  anchored  definitions  for  each  of  three  rating 
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categories.  The  scale  links  ratings  to  the  degree  and  direction  of  statistical  difference  between 
treatment  (DL)  and  control  group  learning  outcomes  as  reported  in  each  of  the  15  studies.  If 
studies  showed  significant  differences  in  favor  of  control  group  learning  outcomes,  a  rating  of  1 
was  assigned.  If  no  statistical  differences  resulted  between  treatment  and  control  groups,  a  rating 
of  2  was  assigned.  Where  differences  were  reported  in  favor  of  the  DL  treatment  group(s)  as 
compared  to  the  control  group,  a  rating  of  3  was  assigned.  While  this  scale  does  not  provide  fine 
gradations  for  differences  in  learning  outcomes,  it  is  clearly  linked  to  the  degree  of  between- 
group  differences  and  eliminates  evaluator  bias  in  arriving  at  ratings. 


DL  Group  Learning  Outcomes  DL  Group  Learning  Outcomes  DL  Group  Learning  Outcomes 

Less  Effective  than  Controls  (p  <0.05)  =  Controls  (p  >0.05)  More  Effective  than  Controls  (p  <0.05) 


Figure  3.  Likert  Scale  used  to  Rate  Learning  Outcome  Effectiveness  for  the  Studies 

The  preliminary  summary  matrix  used  to  classify  DL  studies  is  shown  in  Table  5.  It 
consists  of  DL  methods/media  along  the  vertical  axis  and  skill/knowledge  types  listed 
horizontally.  This  classification  scheme  was  developed  iteratively  with  ARI  review/inputs. 

After  populating  the  Table  5  matrix  with  studies,  many  cells  were  eliminated  due  to  the 
absence  of  studies  applicable  to  those  media  x  skill/knowledge  types.  For  example,  although 
there  is  strong  interest  in  digital  battlefield  skills/knowledge,  no  study  directly  addresses  digital 
proficiency  training.  Other  cells  were  combined  (i.e.,  CMC  and  CBT)  into  a  single  category.  The 
learning  effectiveness  data  were  used  to  produce  histograms  of  rating  frequencies  for  each 
applicable  cell.  That  “raw  data”  matrix  is  included  in  Appendix  C  and  summarized  in  Table  6. 

Table  5  studies  were  reviewed  against  all  possible  skill/knowledge  x  method/media 
combinations,  then  assigned  to  all  applicable  cells  with  a  reference  number.  For  example,  if  a 
study  assessed  higher-order  cognitive  skills  using  video  teletraining  (VTT),  its  reference  number 
was  assigned  to  the  applicable  cell  (VTT  x  Cognitive).  If  the  study  also  appeared  to  include 
procedural  skills,  the  same  study  reference  number  was  assigned  to  the  applicable  cell  (VTT  x 
Procedural). 
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Table  5. 


Preliminary  Matrix  for  Classifying  Studies 


Methods/ 

Media 

Types  of  Learning 

Declarative 

Narrative/ 

Descriptive 

Hands-on/ 

Perceptual- 

Motor 

Procedural 

Cognitive 

Leadership 

Digital 

Affective 

Audio 

Video 

CBT 

Collaborative 

Technologies 

Computer- 

Mediated 

Conferencing 

Intelligent 

Tutoring 

Systems 

Printed 

Materials 

Web-based 

Environments 

Only  one  Table  6  cell  (Audio  x  Perceptual-Motor)  yielded  a  modal  rating  of  3  (DL 
learning  outcomes  statistically  superior  to  controls)  and  that  was  based  on  a  very  small  sample 
(n=3).  All  other  ratings  are  in  the  2  category  indicating  that  DL-trained  groups  learn  as  well  as 
controls.  This  is  consistent  with  Russell’s  (1999)  “no  significant  difference”  finding. 


Table  6. 

Documented  DL  Technology  Effectiveness  for  Training  Skill/Knowledge  Types 


■SBSH 

Audio 

BB31 

Overall  Rating  2 

ES? 

Overall  Rating  2 

ES? 

Overall  Rating  2 

ES  Medium 

Study  No. 

14 

14, 15 

5, 14, 15 

4,  6, 14, 15 

4 

Video 

Overall  Rating  2 

ES  Low 

Overall  Rating  2 

ES? 

Overall  Rating  2 

ES? 

Overall  Rating  2 

ES? 

Overall  Rating  2 

ES  ? 

Study  No. 

9 

2, 3,  8, 10, 

11,12, 13 

I, 2,  3, 5,  7,10, 

II,  12, 13 

1 ,  2,  4,  5,  7, 

9, 10, 11, 13 

1,2,4,9,12,13 

CMC/CBT 

Overall  Rating 

ES 

Overall  Rating 

ES 

Overall  Rating  2 
ES? 

Overall  Rating  2 
ES? 

Overall  Rating  2 

ES? 

Study  No. 

6, 12, 13 

6, 12, 13 

14 

14 


Figure  4  shows  the  frequencies  of  ratings  within  each  category  for  pooled  Table  6  results. 
The  majority  of  the  ratings  (77%)  were  in  the  “no  significant  difference”  (2)  category.  Where 
learning  differences  were  found,  DL-trained  groups  performed  better  than  control  groups 
(Category  3). 
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Figure  4.  Frequency  Distribution  of  Learning  Effectiveness  Ratings 

In  addition  to  the  ratings  of  effectiveness,  we  used  Cohen’s  (1969)  technique  to  compute 
Effect  Sizes  (ES)  for  each  study  reporting  standard  deviations. 

The  ES  is  an  estimate  of  the  treatment  effect  and  is  computed  by: 

Xdl-Xc 
Pooled  SD 

where  X  DL  =  Mean  of  DL  group  measure 

X  c  =  Mean  of  control  group  measure 

and  Pooled  Standard  Deviation  (SD)  can  represent  either  group  since  they  are 
assumed  to  be  equal  (Cohen,  1969). 

Unfortunately,  only  5  of  the  15  studies  in  Figure  2  included  sufficient  data  to  compute  the 
ES.  Using  Cohen’s  guidelines,  we  categorized  ES  strength  as  follows: 

Small  0.50 
Medium  0.51-0.79 
Large  >0.79 

One  study  (2)  reported  SDs  very  small  and  nearly  identical  (overall  SDs  =  0.018-0.025) 
for  treatment  and  control  group  measures.  For  this  study,  mean  test  score  differences  were  only  a 
few  percentage  points.  Although  the  small  SDs  produced  a  large  ES  (5.9),  there  were  no 
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statistically  significant  differences  in  learning  outcomes  between  the  groups.  This  ES  therefore  is 
not  included  in  Table  6. 

Table  6  results  suggest  that  medium  overall  ES  were  found  for  audio  DL  media  in 
training  perceptual-motor  and  leadership  skills.  A  low  ES  resulted  for  the  single  Video  x 
Declarative  study.  No  other  ES  conclusions  are  defensible. 

CURRENT  PROGRAMS 

Following  is  a  brief  synopsis  of  DL  programs  and  initiatives  impacting  those  of  the 
Army’s.  The  plans  have  common  elements,  but  different  focal  areas  and  missions  depending  on 
the  training  needs  of  the  organization. 

Advanced  Distributed  Learning 

Perhaps  the  most  significant  industry-wide  DL  initiative,  the  Advanced  Distributed 
Learning  (ADL)  program,  is  designed  to  encompass  virtually  all  DL-requirements  for  the 
Services,  industry,  academia,  and  the  public  sector.  The  principal  current  focal  area  for  DL  is  the 
development  and  use  of  Reusable  Content  Objects  (RCOs),  Reusable  Learning  Objects  (RLOs), 
and  Knowledge  Objects  (KOs).  The  ADL  initiative,  launched  in  1997  and  cultivated  by  the  DoD 
and  the  White  House  Office  of  Science  and  Technology,  is  aimed  at  the  widespread 
dissemination  of  reusable  content  via  Sharable  Courseware  Object  Reference  Model  (SCORM) 
standards  and  guidelines.  SCORM  ensures  that  learners  have  access  to  standard,  yet  adaptable, 
education  and  training  when  and  where  it  is  needed  using  PC  platforms.  ADL  will  streamline  the 
Instructional  Systems  Development  (ISD)  process  without  compromising  the  integrity  or 
effectiveness  of  emerging  instruction.  ADL  now  uses  “plugfests”  to  test  the  viability  of  the 
content  as  well  as  delivery  speed  and  accuracy.  ADL  success  hinges  on  the  cost  and  learning 
effectiveness  of  emerging  hardware  and  software  products.  Operational  measures  of  success  will 
need  to  be  linked  to  the  ADL  elements.  Further  ADL  information  can  be  found  at 
http://www.adlnet.org.  Additionally,  Parmentier,  Fletcher,  Jesukiewicz,  and  Dodds  (2000) 
provide  a  useful  description  of  ADL  history,  status,  and  projections. 

Naval  Postgraduate  School 

The  DL  Migration  Plan  for  the  Naval  Postgraduate  School  (NPS)  can  be  reviewed  at 
http://web.nps.navv.mil/dlrc/NPS  plan/.  The  plan  is  designed  to  provide  easy  access  to  education 
and  information  for  all  Navy  officers  anytime  and  anywhere.  Extensive  revisions  to  the  curricula 
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and  content  are  being  made  to  achieve  this  goal.  A  vision  of  the  future  war  is  driving  this  effort. 
Some  changes  include  interdisciplinary  emphasis,  integrating  academics  with  military  education, 
and  incorporating  new  information  technologies  for  delivering  selected  instructional  materials. 
The  Warrior  Curriculum  addresses  the  need  to  establish  a  foundation  of  interdisciplinary 
technical  skills  with  a  concentration  of  military  applications  of  information  technology.  The  NPS 
will  offer  a  broad  spectrum  of  courses  that  are  especially  designed  to  raise  the  educational  levels 
of  officers  in  support  of  the  war-fighting  requirements  during  the  next  decade.  The  mission  is 
reflected  in  the  operational  motto,  “from  technical  to  tactical”.  To  facilitate  its  DL  requirements, 
NPS  will  capitalize  on  multimedia,  high-capacity  storage  media,  broad  bandwidth 
telecommunications,  intranets  and  the  Internet,  and  other  methods  designed  to  enhance  student 
learning.  Some  key  objectives  of  this  program  are: 

•  Provide  residential  programs,  as  required 

•  Provide  on-  and  off-campus  instruction  using  traditional  methods/media 

•  Offer  off-campus  video  tele-education  programs 

•  Offer  on-  and  off-campus  instruction  using  portable  media  (CDs  and  DVD) 

•  Provide  off-campus  instruction  using  telecommunication  technologies  (Internet  and 
videoconference) 

•  Facilitate  off-campus  access  to  libraries  via  electronic  media 

The  migration  plan  provides  a  methodology  for  assessing  existing  and  planned  courses 
regarding  appropriateness  for  conversion/development  in  DL  formats.  For  courses  to  be 
converted  to  DL,  the  plan  provides  guidelines  to  technologies  and  criteria  for  evaluating  their 
effectiveness.  A  recommended  migration  process  with  a  timeline  is  included  as  well  as  a  DL 
business  model.  The  NPS  model  is  designed  to  fully  exploit  and  accommodate  the  ADL 
initiative. 

The  Air  Force  DL  Plan 

Two  major  factors  have  driven  DL  within  the  Air  Force — the  need  to  make  content 
readily  available  to  approximately  390,000  learners  located  worldwide  and  the  demands  to  lower 
training  delivery  costs  per  learner.  The  Air  University  reported  savings  of  $700  per  student  using 
satellite  technology.  CD-ROMs  replaced  25  volumes  of  text  to  support  a  correspondence  course 
resulting  in  a  savings  of  $364,000  per  year.  CBT  allowed  learners  to  complete  the  course  in  one- 
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fourth  of  the  time  of  resident  enrollment  and  increased  enrollment  by  almost  500% 

(Cherry,  1996). 

The  Air  Force  Institute  of  Technology  (AFIT)  located  at  Wright-Patterson  AFB,  OH, 
provides  a  full  curriculum  of  master  and  doctorate  degrees,  mainly  in  math  and  sciences.  AFIT 
offers  more  than  20  continuing  education  programs  using  two-way  audio  and  video  teletraining 
via  the  Air  Technology  Network  (ATN)  and  the  Government  Education  and  Training  Network 
(GETN).  ATN  uses  digital  video  and  audioconferencing  for  two-way  audio  interaction  and 
reaches  87  worldwide  sites  (Westfall,  1999).  For  2000,  approximately  9,000  hours  of  DL  were 
delivered  using  ATN  and  GETN. 

The  Air  Force  Distance  Learning  Office  (AFDLO)  has  designed  its  DL  initiatives  around 
the  following  areas  of  concern: 

•  Customer  and  provider  needs  identification 

•  Policies  and  standards  linked  to  AF  requirements 

•  Information  dissemination  throughout  the  AF 

•  Data  collection,  analysis,  and  comprehensive  reporting 

•  Administrative  requirements  for  tracking  student  and  DL  courseware  data 

•  Scheduling  and  coordination  of  the  Air  Technology  Network  for  delivery 

•  Assigning  program  managers  and  manpower  for  ATN  management 

Information  on  the  Air  Force  Institute  for  Advanced  Distributed  Learning  is  available  at 

http://www.au.af.mil/au/afiadl. 


CONCERNS  AND  CAVEATS 

TRW  has  provided  a  DL  research  product  that  is  in  compliance  with  our  Statement  of 

Work.  Two  concerns  apply  to  the  use  of  the  Table  6  matrix  for  assessing  DL  effectiveness.  First, 

it  provides  a  somewhat  over-simplified,  two-dimensional  view  of  a  complex  issue  (DL 

effectiveness).  The  literature  is  clear  on  the  multidimensional  nature  of  DL.  The  prevailing  view 

is  that  DL  effectiveness  stems  not  from  methods/media/technology,  but  from  a  conscientious  and 

deliberate  attempt  to  design,  develop,  deliver,  manage,  and  evaluate  instruction  from  a  systems 

viewpoint.  According  to  Clark  (1983), 

“...the  best  current  evidence  is  that  media  are  mere  vehicles  that 
deliver  instruction  but  do  not  influence  student  achievement 
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(learning  outcomes )  any  more  than  the  truck  that  delivers  our 
groceries  causes  changes  in  our  nutrition.  ” 

Clark  (1983)  further  states  that: 

“...media  are  still  advocated  for  their  ability  to  increase  learning  when 
research  clearly  indicates  that  such  benefits  are  not  forthcoming. 

Of  course  such  conclusions  are  disseminated  slowly  and  must 
compete  with  advertising  budgets  of  the  multi-million  dollar 
industry  which  has  a  vested  interest  in  selling  machines.  ” 

While  the  above  comments  were  published  well  before  the  advent  of  the  Internet,  they 
are  perhaps  even  more  valid  now  than  nearly  two  decades  ago.  Guice  (1997)  has  indicated  that 
virtual  classrooms  have  not  sufficiently  addressed  social  and  educational  questions.  He  states 
that, 

“...too  much  attention  is  focused  on  hardware  and  software.  Too 
little  attention  is  invested  in  how  the  technologies  are  to  be  used, 
and  how  media  affect  the  experiences  that  students  have.  Many 
leading  efforts  now  attempt  to  put  people  and  learning,  rather  than 
technology  in  itself,  at  the  center  of  attention.  ” 

Joy  and  Garcia  (2000)  indicate  that  practitioners  should  not  assume  that  students  learn 
better  from  technology-based  systems,  but  rather  that  instructional  design  strategies  control 
learning  effectiveness,  regardless  of  medium.  They  examine  five  randomly  chosen  studies  and 
report  significant  design  flaws  in  each  of  them. 

Our  second  concern  lies  with  assigning  ratings  to  skill/knowledge  type  based  solely  on  a 
review  of  the  study  methodology.  Some  studies  include  very  sketchy  information  on  the  specific 
types  of  skills/knowledge  addressed,  leaving  the  reader  to  infer  skill/knowledge  types  from  brief 
generic  descriptions  of  the  course(s).  For  example,  Study  10  (Wetzel,  1996)  lists  a  Celestial 
Navigation  course  as  one  of  the  targeted  topics  for  DL  effectiveness  assessments.  The  study, 
however,  provides  no  description  of  the  specific  knowledge  or  performance  requirements  for  the 
course.  To  attenuate  concerns  based  on  incomplete  information  in  many  of  the  studies,  we 
decided  to  devise  a  rating  procedure  that  minimizes  subjectivity.  Our  procedure  used  a  Likert 
scale  with  clearly  anchored  definitions  for  each  rating  category.  The  scale,  shown  in  Figure  3, 
links  ratings  to  the  degree  and  direction  of  statistical  difference  between  treatment  (DL)  and 
control  group  learning  outcomes  as  reported  in  each  study. 

Finally,  in  regard  to  experimental  design  issues,  it  should  be  noted  that  DL  studies  can 
unintentionally  bias  study  conditions  by  using  more  resources,  more  highly  motivated 
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instructors,  mentoring/coaching,  greater  incentives  for  learners,  or  better  planning  for  DL 
treatment  groups  as  compared  to  the  control  groups.  There  should  be  concerted  efforts  to  ensure 
a  match  between  learning  objectives  and  outcome  tests  of  those  objectives.  Ironically,  in 
attempting  to  address  the  very  issues  that  make  DL  more  effective  and  efficient,  we  may 
confound  DL  delivery  in  favor  of  the  treatment  group,  thereby  leading  to  misinterpretation  of 
results. 


CONCLUSIONS 

Table  6  data  are  useful  for  summarizing  the  categories  of  methods/media  that  are 
currently  used  to  train  the  various  DL  skill/knowledge  types.  However,  the  matrix  does  not  offer 
practical  help  for  assigning  media  to  skill/knowledge  types  because  ratings  and  effect  sizes  are 
not  discriminators  in  the  classification  scheme.  It  is  therefore  intended  as  a  tool  that  can  be  used 
to  incorporate  additional  factors  (e.g.,  cost,  resource  use,  user  acceptability,  technical  constraints) 
as  the  DL  database  grows.  Indeed,  a  multidimensional  matrix  that  allows  users  to  query  the 
database  relative  to  the  DL  variable  of  interest  should  be  a  goal.  An  electronic  wizard  that 
provides  quantitative  and  qualitative  guidance  on  the  design  of  DL  environments  is  one 
outgrowth  of  such  a  goal. 

Consistent  with  Russell’s  (1999)  results,  Table  6  ratings  indicate  that  the  DL  media 
employed  within  empirical  investigations  are  statistically  as  effective  as  conventional  training  in 
producing  the  desired  learning  outcomes.  This  suggests  that,  in  addition  to  its  more  widely 
disseminated  efficiency  benefits,  DL  is  promising  for  achieving  learning  effectiveness.  But  it  is 
not  the  DL  media  that  yield  the  effectiveness.  When  employed  with  sound  instructional  design, 
purposeful  mentoring  and  guidance  by  instructors,  and  a  genuine  desire  on  the  part  of  students 
to  learn,  DL  will  likely  be  successful.  The  literature  clearly  supports  this  conclusion  (Clark,  1983; 
Clark  and  Salomon,  1986;  Phipps  and  Merisotis,  1999;  Wisher  et  al.,  1999;  Machtmes  and  Asher 
2000;  and  Champagne  and  Wisher,  2000).  Regardless  of  whether  students  are  separated  by 
teaching  sources  in  space  or  time,  decisions  about  the  use  of  various  training  media  should  be 
made  on  the  basis  of  factors  other  than  learning  effectiveness.  Instructors  must  feel  comfortable 
with  their  role  in  guiding  learners  through  the  DL  process  to  reduce  attrition  and  facilitate 
learning  (Carr,  2000).  Learner-centered  approaches  link  training  content  to  the  needs  and 
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experience  of  the  learners  (Hannafin  and  Land,  1997).  However,  such  approaches  are  seldom 
possible  in  military  settings. 

Although  empirical  in  nature,  Table  6  studies  generally  lack  statistical  detail  required  to 
calculate  accurate  estimates  of  effect  sizes.  The  evaluation  factors  included  in  Figure  2  are 
recommended  as  guidelines  for  designing  and  conducting  effective  DL. 

The  Advanced  Distributed  Learning  program  is  expected  to  standardize  DL  requirements 
for  the  military  services  as  well  as  industry,  academia,  and  the  public  sector.  As  the  ADL 
program  becomes  more  mature  and  standardized,  more  valid  comparisons  of  DL  attributes 
should  be  possible. 
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APPENDIX  A  -  NON-EMPIRICAL  DL  RESEARCH  SUMMARY 


NON-EMPIRICAL  DL  RESEARCH  SUMMARY 


Author/Reference 

Summary 

Alexander (1999) 

This  article  describes  a  “collaborative  instructional  design  project 
using  constructivist  theory,  exploratory  and  resource-based 
learning,  electronic  communities,  and  integrated  information 
technology  immersion.”  Learning  outcomes  for  student  and 
faculty  participants  in  a  graduate  health  services  planning  and 
policy  course  are  discussed  and  analyzed. 

Bond  &  Pugh  (2000) 

This  paper  describes  distance  learning  training  attempts  for  two 
Army  National  Guard  courses:  Air  Traffic  Control  and  Helicopter 
Maintenance.  General  lessons  learned  are  noted. 

Bonk  &  Wisher  (2000) 

Research  that  adapts  and  evaluates  new  collaborative  and  e- 
leaming  tools  in  Army  settings  can  determine  whether  the 
benefits  of  innovative  approaches  to  education  can  also  benefit 
training.  This  article  recommends  ten  primary  experiments  to 
serve  as  a  research  framework  to  adapt  and  tailor  e-leaming 
technologies  to  Army  training  needs. 

Brown  (1996) 

The  Army  Distance  Learning  Plan  addresses  DL  applications  and 
directs  major  commands  to  develop  DL  plans  to  meet  their 
training  requirements.  The  Medical  Command’s  U.S.  Army 
Medical  Department  DL  Plan,  addresses  the  overall  requirements 
of  DL  application  for  medical  training.  As  a  result  of  this 
progression,  conversion  of  combat  medical  training  into  distance 
learning  format  has  become  a  priority  for  continued  readiness  in 
the  U.S.  Army  Medical  Department. 

Brown  &  Wack  (1999) 

The  authors  question/critique  the  report  “What’s  the  Difference?” 
(Phipps  and  Merisotis,  1999),  prepared  by  The  Institute  for 

Higher  Education  Policy.  The  authors  conclude  that,  while  more 
and  better  assessment  of  distance  learning  is  needed,  what’s 
needed  even  more  is  for  that  research  to  inform  practice. 

Buscho  &  Knutson  (1999) 

This  paper  describes  the  Advisor  Learning  Network  provided  for 
American  Express  Financial  Advisors.  Extensive  training  is 
provided  on  a  wide  variety  of  topics  using  a  variety  of  methods. 

Calder  (2000) 

Regarding  the  development  and  provision  of  best  practice  and  the 
place  of  critical  reflection  by  stakeholders,  this  article  considers 
what  some  contributors  to  the  literature  on  open  and  distance 
learning  have  to  say. 

Champagne  &  Wisher 
(2000) 

Provides  the  practitioner  with  an  understanding  of  the  essential 
elements  of  a  beneficial  evaluation,  with  an  emphasis  on 
considerations  for  evaluation  design. 

A-1 


Author/Reference 

Summary 

Champagne,  Wisher, 

Pawluk,  &  Cumow  (1999) 

This  paper  presents  a  critical  review  of  empirical  studies  designed 
to  compare  training  by  distance  learning  technologies  to  training 
by  traditional  classroom  instruction. 

Crawford  &  Suchan  (1996) 

This  report  addresses  the  use  of  instructional  media  in  teaching 
executive  management  education.  The  research  proposes  four 
learning  outcomes  and  examines  them  in  the  context  of  the 
instructional  techniques  required  to  support  them. 

Cyrs  (1997) 

The  authors  of  the  book  are  among  those  who  believe  that 
distance  education  is  the  future  of  education,  and  institutions  that 
will  survive  are  those  who  use  the  technology  to  deliver  programs 
that  are  convenient  and  cost-effective  for  learners.  Timely 
explanations  for  experienced  distance  teachers  and  practitioners 
offer  rich  information  about  skills  needed  for  distance  education. 

Despain  (1997) 

This  study  addressed  the  integration  of  computer-delivered 
listening  comprehension  exercises  into  the  university-level 
foreign  language  curriculum.  Results  suggest  that  students  tend  to 
learn  more  effectively/efficiently  using  the  computer  delivery 
system. 

Dodge,  Webb,  &  Christ 
(1999) 

This  report  reviews  and  analyzes  the  individual  and  collective 
effects  of  information  technology  as  presented  in  the  management 
science  and  business  literature.  The  goal  of  the  report  was  to 
address  the  impact  of  information  technology  on  the  human 
dimensions  of  battle  command. 

Douzenis  (1998) 

i 

The  purpose  of  the  study  presented  in  this  paper  was  to  determine 
the  relationship  between  cognitive  style  and  achievement  in  a 
master’s  level  educational  research  course  taught  using  distance 
education  technology.  Results  of  the  study  demonstrated  a 
relationship  between  cognitive  style  measures  and  achievement  in 
a  distance  education  course.  However,  only  20%  of  the  variance 
in  achievement  was  explained  by  these  variables.  Limitations  of 
this  study  include  a  small  sample  size  and  measurement  of 
achievement  in  only  one  course  subject  (educational  research). 

Goldberg  (1997) 

This  paper  describes  an  experiment  evaluating  Web-based  course 
delivery  in  terms  of  academic  performance  and  student 
acceptance.  Students  in  a  third-year  university  computer  science 
course  were  divided  into  three  groups  (Web  only,  lecture  only, 
and  combined  Web/lecture).  In  general,  the  Web-only  delivery 
was  equally  successful  compared  to  the  lecture-only  delivery.  On 
average,  students  in  the  combined  lecture/Web  delivery  group 
performed  academically  better  than  students  in  the  other  group. 

Author/Reference  _ Summary _ 

Green  (1997)  “Begun  in  1990,  the  annual  Campus  Computing  Project  is  the 

largest  continuing  study  of  information  technology  in  American 
higher  education.”  The  1999  survey  data  were  provided  by 
campus  officials  (typically  the  senior  technology  officer)  at  557 
2-  and  4-year  public  and  private  colleges  and  universities  in  the 
U.S. 

Guice  (1997)  This  article  describes  and  evaluates  a  seminar  course  taught 

jointly  between  Stanford  University  and  Sweden’s  Royal  Institute 
of  Technology  using  videoconferencing  over  the  Internet.  The 
author  reports  that  curriculum  and  instructional  design  are  more 
important  than  technology  for  distance  learning  success. 

Hassen,  Wieckhorst,  This  report  documents  the  structured  approach  developed  for 

Madden,  Franz,  &  Dunlap  reviewing  formal  Navy  courses  to  identify  opportunities  to  reduce 

(1996)  training  time  and  costs  through  the  application  of  advanced 

training  technologies.  This  effort  was  complicated  by  the 
“unmanageably  large  numbers”  of  courses  to  be  reviewed.  The 
Training  Delivery  Assessment  Model  (TRAD AM)  performed 
extremely  well  during  training  efficiency  review  of  40  formal 
training  courses. 

Hiltz,  Coppola,  Rotter,  This  paper  describes  three  studies  (field  study,  field  experiment, 

Turoff,  &  Benbunan-Fich  and  semi-structured  interviews)  that  address  the  importance  of 

(2000)  collaborative  learning  strategies  to  the  success  of  ALN  for 

students.  A  study  of  26  courses,  part  of  an  undergraduate 
Information  Systems  program,  compared  the  process  and 
outcomes  of  learning  using  an  on-line  anytime/anywhere 
environment  to  those  sections  taught  in  the  classroom.  Results 
support  the  premise  that  when  students  are  actively  involved  in 
collaborative  (group)  learning  on-line,  the  outcomes  are  as  good 
as  or  better  than  those  for  traditional  classes.  But  when  students 
are  just  receiving  posted  material  and  returning  individual  work, 
the  results  are  poorer  than  that  for  classrooms. 

Jewett  (1997)  This  report  is  one  of  a  series  from  a  project  entitled  Case  Studies 

in  Evaluating  the  Benefits  and  Costs  of  Mediated  Instruction  and 
Distributed  Learning.  The  subject  of  the  case  study  is  a  graduate 
level  certificate  program  in  human  computer  interaction.  All 
courses  are  new  and  designed  for  delivery  in  a  distributed 
learning  environment.  For  the  course  evaluated,  there  was  little 
variation  among  final  grades  for  students  enrolled  in  the  three 
modes  (on  campus,  videoconferencing  groups,  and  videotape 
groups.) 
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Author/Reference 

Jewett  (1998a) 


Jewett  (1998b) 


Johns (1999) 


Jones  &  Paolucci  (1998) 


Joy  &  Garcia  (2000) 


Summary 

This  report  is  one  of  a  series  from  a  project  entitled  Case  Studies 
in  Evaluating  the  Benefits  and  Costs  of  Mediated  Instruction  and 
Distributed  Learning.  The  University  of  Akron  and  Cleveland 
State  University  are  offering  a  joint  graduate  program  leading  to 
the  Master  of  Social  Work  degree  using  interactive 
videoconferencing.  There  is  no  evidence  of  a  difference  in 
learning  outcomes  (measured  by  grades)  for  students  at  the 
sending  or  receiving  sites. 

This  report  is  one  of  a  series  from  a  project  entitled  Case  Studies 
in  Evaluating  the  Benefits  and  Costs  of  Mediated  Instruction  and 
Distributed  Learning.  A  philosophy  course  was  redesigned  using 
a  combination  of  synchronous/asynchronous  methods.  One 
section  was  taught  in  the  “interactive  web-based  mode”  and  the 
other  section  was  taught  in  a  more  traditional  large  lecture/small 
discussion  format.  Evidence  regarding  learning  outcomes  is 
“somewhat  ambiguous.”  A  microbiology  course  was  restructured 
to  improve  student  access  to  course-related  materials  and  to 
improve  communications  between  students  and  faculty  and 
among  students  (bottom  row  at  right).  This  was  accomplished 
through  the  creation  of  an  elaborate  course  Web  site  created  to 
allow  student  access  to  class  announcements,  documents,  practice 
exams,  chat  rooms,  message  boards,  etc.  The  grades  provided  no 
evidence  that  learning  outcomes  changed  as  a  result  of  the 
network  technology  (Web  site).  (Goal  of  redesign  was  to  improve 
communication,  not  to  change  grades.) 

This  paper  demonstrates  3-D  practice  environments  developed  to 
teach  mechanical  skills.  The  entire  multi-layered  application  can 
be  delivered  via  Web  browsers  on  a  corporate  LAN  or  on  the 
Internet. 

The  authors  estimate  that  less  than  5%  of  published  research  is 
sufficiently  empirical,  quantitative,  and  valid  to  support 
conclusions  with  respect  to  the  effectiveness  of  technology  in 
educational  learning  outcomes.  The  authors  conclude  with  a  call 
for  further  research  concentrating  on  the  application  of 
appropriate  technologies  to  the  learning  outcomes  of  the  subject 
matter  to  which  technology  is  applied. 

This  paper  outlines  the  philosophical  positions  of  the  opposing 
sides  in  the  literature  as  to  whether  delivery  media  alone 
influences  learning  outcomes.  Several  representative  media 
comparison  studies  were  selected  at  random  to  illustrate  the 
inadequacy  of  their  methodologies  and  conclusions.  ALN 
practitioners  should  adhere  to  time-tested  instruction  design 
strategies,  regardless  of  the  medium  chosen. 


Author/Reference 

Summary 

Kribs  &  Mark  (1998) 

The  objective  of  this  demonstration  project  was  to  determine 
whether  VTT  technology  could  be  integrated  with  automated 
electronic  classroom  (AEC)  technology  to  deliver  computer- 
based  performance  training.  The  results  indicate  that  the 
combination  of  VTT  and  AEC  can  be  used  to  successfully  deliver 
computer-based  performance  instruction. 

Kronholm,  Wisher,  Cumow, 
&  Poker  (1999) 

This  paper  describes  the  steps  being  taken  to  transform  the 
advertising,  scheduling,  enrollment,  and  evaluation  of  satellite- 
delivered,  short-term  training  from  dependence  on  the  telephone, 
fax,  mail,  and  paper  and  pencil  forms  to  reliance  on  the  Internet 
as  a  value-added  resource. 

Leh (1999) 

This  paper  examines  research  conducted  on  computer-mediated 
communication  (CMC)  in  foreign  language  learning.  The  study 
revealed  that  CMC  was  beneficial  for  distance  learning.  A 
follow-up  study  was  conducted  1  year  after  the  initial  study  was 
completed.  The  results  of  the  follow-up  study  supported  the 
findings. 

Lewis,  Snow,  Farris,  & 

Levin  (1999) 

This  report  presents  findings  from  the  second  nationally 
representative  survey  of  distance  education  undertaken  by  the 
National  Center  for  Educational  Statistics.  The  survey  collected 
information  about  the  12-month  1997-98  academic  year.  This 
report  provides  trend  information  for  higher  education 
institutions,  including  changes  in  the  percentage  of  institutions 
offering  distance  education  courses,  enrollments  and  course 
offerings,  degree  and  certificate  programs,  and  technologies  used 
to  deliver  distance  education  courses. 

Liu  &  Ginther  (1999) 

This  paper  addresses  the  issue  of  adapting  the  design  of  distance 
education  to  students’  cognitive  styles. 

Mitchell  (1996) 


Navarro  &  Shoemaker 
(2000) 


This  paper  examines  the  concept  and  theory  of  distance  learning, 
briefly  traces  the  history  of  its  development,  and  describes 
technology  currently  available.  The  paper  discusses  issues  of 
quality  and  institutional  planning  and  management  and  suggests 
some  potential  applications  at  the  Army  War  College.  Through 
distance  learning,  the  Army  War  College  core  missions  can  be 
enhanced  to  meet  the  challenges  of  increased  need  for  strategic 
knowledge  at  all  levels,  limited  or  declining  resources,  and  the 
changing  needs  of  students. 

This  article  describes  a  study  of  several  hundred  students  in  an 
introductory  economics  course.  The  cyberleamers  learned  as  well 
as,  or  better  than,  traditional  learners,  regardless  of  such 
characteristics  as  ethnicity,  gender,  computer  skills,  and  academic 
background/aptitude. 


,-5 


Author/Reference 

Summary 

Neal  (1997) 

This  paper  describes  how  collaborative  learning  technologies  can 
be  used  to  teach  classes  to  geographically  dispersed  participants. 
The  paper  covers  the  motivation  for  virtual  classrooms,  the 
selection  and  use  of  delivery  technologies,  deployment  strategies 
and  issues,  participant  feedback,  and  the  Virtual  University  that 
evolved  from  the  initial  distance  learning  classes.  The  study 
found  that  the  use  of  a  variety  of  collaborative  technologies 
accommodated  the  multiple  aspects  of  communication  in  the 
class. 

Neal  &  Ingram  (2000) 

This  paper  describes  the  use  of  Lotus  LeamingSpace  to  teach  an 
asynchronous  distance  learning  class  on  human-computer 
interaction  in  a  corporate  setting.  Approaches  are  needed  that  will 
manage  both  teachers’  and  learners’  expectations  about 
asynchronous  instruction,  while  highlighting  the  advantages  of 
this  delivery  method,  as  well  as  methods  and  strategies  that  will 
enable  teachers  to  more  effectively  develop  courses  for 
asynchronous  delivery. 

Oblinger  &  Maruyama 
(1996) 

The  information  in  this  paper  captures  “the  rationale  for  the 
growing  movement  to  deploy  networking  technologies 
strategically  in  instruction.”  The  authors  “articulate  the  need  for 
higher  education  to  create  affordable  and  flexible  student- 
centered  ‘distributed  learning  environments’  which  differ  in 
fundamental  ways  from  today’s  teacher-centered  classrooms.” 

Ory,  Bullock,  &  Bumaska 
(1997) 

This  paper  presents  the  results  of  an  investigation  of  male  and 
female  student  use  of  and  attitudes  about  ALN  after  one  year  of 
implementation  in  a  university  setting.  Results  of  the  study 
revealed  no  significant  gender  differences. 

Parker  (1999) 

A  need  exists  for  research  to  determine  predictors  of  dropout 
from  distance  education  since  attrition  rates  in  distance  education 
far  exceed  rates  in  classes  taught  in  a  traditional  format.  The 
results  of  the  study  presented  in  this  paper  are  important  as 
counselors  and  faculty  now  have  a  basis  on  which  to  advise 
students  either  into  traditional  or  distance  formatted  courses  for 
best  possible  completion. 

Perraton  (2000) 

This  paper  advances  four  propositions  about  the  links  between 
theory  and  good  practice  and  follows  the  propositions  with  four 
proposals  about  important  areas  of  research. 

Author/Reference  Summary 

Phipps  &  Merisotis  (1999)  The  Institute  was  commissioned  by  the  American  Federation  of 

Teachers  and  the  National  Education  Association  to  review  the 
current  research  on  the  effectiveness  of  distance  education  and  to 
analyze  what  the  research  says  and  does  not  say.  The  report 
suggests  that  too  many  of  the  questions  posed  are  left 
unaddressed  or  unanswered  in  the  research. 

Phipps,  Wellman,  &  The  authors  were  commissioned  to  “investigate  the  emerging 

Merisotis  (1998)  topic  of  quality  assurance  in  technology-mediated  distance 

learning  programs  in  higher  education.” 

Rossman  (1999)  A  document  analysis  of  more  than  3,000  course  evaluations  from 

154  courses  conducted  during  11  quarters  was  conducted. 
Narrative  responses  from  course  folders  were  grouped  into 
categories.  General  observations  related  to  the  categories  were 
presented,  followed  by  several  tips  for  successful  teaching  in  an 
online  environment  using  an  asynchronous  learner  discussion 
forum. 

Saba  (2000)  The  purpose  of  this  article  is  to  present  a  “coherent  view  of  the 

state  of  research  in  distance  education.”  The  studies  revealed  the 
“complexity  of  distance  education,  indicating  the  many  variables 
involved  in  any  instructional  setting...” 

Schar,  Schluep,  Schierz,  &  Five  experiments  were  performed  in  order  to  investigate  the 

Krueger  (2000)  effect  of  the  computer  user-interface  on  learning  performance. 

The  theoretical  motivation  was  to  validate  the  relevance  of  a 
cognitive  theory  about  two  modes  of  learning  in  a  human- 
computer  interaction  context.  The  experiments  showed 
consistently  that  the  two  learning  modes  can  be  induced  by 
different  user-interfaces,  and  that  the  induced  learning  mode  has 
an  effect  on  the  learning  performance. 

Schutte  (1996)  This  report  describes  two  methods  used  to  present  a  social 

statistics  class:  traditional  classroom  and  on  the  World  Wide  Web 
(virtual  class).  The  virtual  class  scored  an  average  of  20%  higher 
than  the  traditional  class  on  examinations. 

Stoney  &  Oliver  (1999)  The  study  outlined  in  this  paper  was  part  of  a  larger  study  that 

examined  the  use  of  interactive  multimedia  (multimedia 
microworld)  in  motivating  and  engaging  adult  learners.  The  study 
found  that  using  a  microworld  meant  that  more  emphasis  was 
placed  on  the  acquisition  of  higher-order  thinking  and  problem¬ 
solving  skills,  with  less  emphasis  placed  on  the  assimilation  of  a 
large  body  of  isolated  facts. 


Author/Reference 


Stouffer  (1998) 

Strauss  &  Frost  (1999) 

Summerville  (1999) 

Thach (1996) 

Throne  &  Lickteig  (1997) 

Trindale,  Carmo,  &  Bidarra 
(2000) 

Wagner  (2000) 


Summary 

The  major  elements  of  information  that  senior  managers  need  to 
know  in  planning  distributed  training  are  the  cornerstone  of  the 
database.  A  significant  amount  of  information  has  been  compiled 
in  the  database,  enabling  it  to  answer  specific  training-related 
questions:  How  much  goes  on  where?  With  whom?  To  what  end? 
With  additional  budget-related  data,  the  database  could  be  a 
powerful  informational  tool. 

Nine  key  factors  influence  instructional  technology  media 
selection  at  the  university.  After  describing  various  dimensions  of 
the  factors,  the  authors  present  two  conceptual  guides  to  assist  in 
selecting  technology  for  marketing  classes,  focusing  on  cognitive 
and  skill-based  learning  objectives. 

The  focus  of  this  article  was  to  examine  the  variables  of  cognitive 
style,  subject  awareness  of  the  instructional  implications  of 
cognitive  style,  and  matching/mismatching  subjects  with 
cognitive  style.  These  variables  may  be  important  in  the  design  of 
instructional  environments,  such  as  hypermedia,  adapted  to 
accommodate  individual  differences. 

Defines  distance  learning  and  associated  terms  and  explains  its 
benefits.  Describes  how  to  get  started  beginning  with  the  analysis 
phase  and  moving  through  the  instructional  design  process,  who 
should  be  involved  in  the  development  team,  delivery  tips,  and 
evaluation  methods.  Job  aid  is  an  ROI  worksheet. 

The  bibliography  assembles  available  literature  bn  computer 
skills  training  from  commercial,  educational,  and  military 
domains.  The  review  consolidates  the  referenced  literature  by 
extracting  key  lessons  learned  categorized  by  the  acquisition, 
retention,  and  transfer  of  computer  skills. 

This  article  presents  a  selection  of  examples  that,  in  the  authors’ 
words,  “the  chosen  cases”  should  not  be  “considered  as  clearly 
better  than  any  other  one,  nor  missing  cases  be  interpreted  as  lack 
of  appreciation  or  a  negative  judgement.” 

The  paper  suggests  that  emerging  e-leaming  models  will 
increasingly  be  based  on  a  reusable  learning  object  architecture 
paradigm. 
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Author/Reference 


Summary 


Walsh,  Gibson,  Miller, 

Hsieh,  Gettman,  & 

Newcomb  (1996) 

The  reported  research  effort  focused  on  the  impact  of  distance 
learning  on  the  curriculum,  types  of  student-instructor  interaction, 
student  interaction  with  the  instmctional  materials,  and  on  the 
preparation  of  faculty  and  staff  for  conducting  distance  learning. 
The  report  also  details  the  development,  composition,  and 
distribution  of  a  distance  learning  survey  and  summarizes  the 
results  associated  with  the  data  analysis. 

Walters  &  Reed  (1997) 

The  report  describes  two  methods  used  to  present  an  introductory 
computing  class:  conventional  lectures/laboratories  and 
independent  study.  Results  based  on  pre-  and  post-testing  of  both 
groups  show  that  independent  study  students  performed  as  well 
as  those  in  the  lecture  course. 

Wegner,  Holloway,  & 

Garton  (1999) 

Graduate  students  in  a  curriculum  design  and  evaluation  course 
were  involved  in  a  two-semester  study  of  the  effects  of  distance 
learning  on  student  achievement.  No  significant  differences 
between  the  test  scores  of  the  Internet-based  test  group  and  in- 
class  group  were  found. 

Wisher  (2000) 

This  paper  examines  the  following  issues:  how  well  soldiers  can 
learn  through  distance  learning  technologies,  the  special 
requirements  for  measuring  performance,  limiting  factors,  and 
updating  training  policies. 

Wisher  &  Cumow  (1998) 

This  report  describes  an  approach  to  creating  a  simplified  form 
for  evaluating  the  relevance  and  effectiveness  of  a  distance 
learning  event  (defined  as  a  training  or  educational  program 
occurring  within  1  day). 

Wisher,  Sabol,  &  Ellis 
(1999) 

This  report  reviews  what  is  known  about  forgetting  as  it  applies 
to  military  tasks,  concentrating  on  major  projects  by  the  U.S. 

Army  Research  Institute.  This  review  makes  clear  several  ways 
that  the  Army  can  minimize,  or  reverse,  the  effects  of  forgetting. 

Wisher,  Champagne, 

Pawluk,  Eaton,  Thornton,  & 
Cumow  (1999) 

! 

This  report  offers  a  review  of  the  literature  on  the  effectiveness  of 
distance  learning  as  applied  to  training.  An  assessment  of  the 
experimental  designs,  reporting,  and  interpretability  of  the 
findings  was  needed  as  previous  reviews  focused  on  whether 
there  was  “no  significant  difference”  between  distance  learning 
and  classroom  comparison  groups.  The  authors  found  that  the 
research  literature  “focused  on  education  rather  than  training,  was 
largely  anecdotal,  and  when  effectiveness  was  examined,  it  was 
not  supported  by  strong  experimental  or  quasi-experimental 
design.” 
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APPENDIX  B  -  DEFINITIONS  AND  GLOSSARY 


METHODS  AND  MEDIA 


Method/Media 

Definition/Delivery  Method 

Audio 

Delivered  over  cassette  players,  PCs,  telephone,  radio,  or 

Internet:  audio  cassettes/conferencing/teletraining,  CD-ROM, 
radio  broadcast,  streaming  audio,  voice  mail 

CBT 

Delivered  through  stand-alone  training  applications,  audio  and 
video:  CD-ROM,  computer  peripherals,  mass  storage  devices, 
printers,  etc. 

Collaborative  Technologies 

Two  or  more  people  working  together  through  electronic  means 
(student-to-student),  group  process,  mentoring;  asynchronous 
and  synchronous 

Computer-Mediated 

Conferencing 

Delivered  through  computer  networks:  application  sharing, 
audiographics,  bulletin  board,  white  board,  and  chat  room  and 
e-mail  (instructor-to-student) 

Intelligent  Tutoring  Systems 
(Computational  Intelligence) 

“Intelligent  tutors  represent  advanced  forms  of  cognitive 
technologies  that  use  computational  intelligence.  They  are 
designed  to  provide  adaptive  instruction  to  users  in  the  effort  to 
promote  and  develop  expert-like  problem-solving  skills.  ...very 
useful  in  reducing  the  time  required  by  learners  to  acquire 
expert-like  problem-solving  skills  that  would  normally  take  two 
to  three  times  as  long  to  develop  using  more  traditional 
instructional  approaches.  ...one  of  the  most  important 
instructional  techniques  offered  by  an  ITS  tutor  is  to  assess  the 
student’s  readiness  to  leam  and  then  to  bring  to  the  student’s 
awareness  relevant  prior  knowledge  and  then  to  help  clarify  the 
relationship  between  the  new  and  old  learning:  to  understand  the 
relationships  that  exist  among  concepts  or  principles  that  lead  to 
better  performance.”1 

Printed  Materials 

Delivered  through  mail,  FAX,  or  Internet  downloads: 
correspondence  study,  training  manuals,  study  guides 

Video 

Delivered  over  videocassette  players,  PC,  satellite,  microwave, 
fiber  optic,  cable,  telephone,  or  Internet:  teletraining,  CD-ROM, 
DVD,  streaming  video;  videocassette 

Web-based  Environments 

1-way  video,  1-way  audio;  2-way  video,  2-way  audio;  CD- 
ROM;  DVD;  streaming  video 

1  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 
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LEARNING  STYLES 


Learning  Style 

Definition/Examples 

Affective  Learning 

(Bloom’s 

Taxonomy)2,3 

Demonstrated  by  behaviors  indicating  attitudes  of  awareness,  interest, 
attention,  concern,  and  responsibility;  ability  to  listen  and  respond  in 
interaction  with  others;  and  ability  to  demonstrate  those  attitudinal 
characteristics  or  values  which  are  appropriate  to  the  test  situation  and 
the  field  of  study.  The  five  levels  within  this  domain  (with  outcome- 
illustrating  verbs)  are  listed  below. 

(1)  Receiving.  The  student’s  willingness  to  attend  to  particular  stimuli. 
The  instructor  must  be  able  to  get,  hold,  and  direct  the  student’s 
attention.  Learning  outcomes  range  from  the  simple  awareness  that  a 
thing  exists  to  selective  attention  on  the  part  of  the  learner.  Asks, 
chooses,  describes,  erects,  follows,  gives,  holds,  identifies,  locates, 
names,  points  to,  replies,  selects,  sits,  uses. 

(2)  Responding.  Active  participation  on  the  part  of  the  student.  They  not 
only  attend  to  a  particular  phenomenon,  but  also  react  to  it  in 
someway.  Learning  outcomes  may  emphasize  compliance  in 
responding,  willingness  to  respond,  or  satisfaction  in  responding. 

The  higher  levels  of  responding  include  those  in  which  the  student 
shows  interest.  Answers,  assists,  complies,  conforms,  discusses, 
greets,  helps,  labels,  performs,  practices,  presents,  reads,  recites, 
reports,  selects,  tells,  writes. 

(3)  Valuing.  The  worth  or  value  a  student  attaches  to  a  particular  object, 
phenomenon,  or  behavior.  This  ranges  from  the  simple  acceptance  of 
a  value  to  the  more  complex  level  of  commitment.  Valuing  is  based 
on  the  internalization  of  a  set  of  specified  values,  while  clues  to  these 
values  are  expressed  in  the  student’s  overt  behavior  and  are  clearly 
identifiable.  Objectives  dealing  with  students’  attitudes  and 
appreciation  would  fall  into  this  category.  Completes,  describes, 
differentiates,  explains,  follows,  forms,  initiates,  invites,  joins, 
justifies,  proposes,  reads,  reports,  selects,  shares,  studies,  works. 

(4)  Organization.  Contrasting  different  values,  resolving  conflicts 
between  them,  and  organizing  a  unique  value  system.  The  emphasis 
is  on  comparing,  relating,  and  synthesizing  values.  Learning 
outcomes  may  be  concerned  with  the  conceptualization  of  a  value  or 
with  the  organization  of  a  value  system.  Adheres,  alters,  arranges, 
combines,  compares,  completes,  defends,  explains,  generalizes, 
identifies,  integrates,  modifies,  orders,  organizes,  prepares,  relates, 
synthesizes. 

2  Bloom’s  taxonomy,  http://www.tecweb.org/eddevel/blooms.html 

3  Taxonomy  of  educational  objectives.  http://www.usd.edu/admin/vDaa/assessment/taxonomies.html 
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Learning  Style 


Definition/Examples 


(S')  Characterization  bv  a  Value  or  Value  Complex.  For  a  student  to 
develop  a  characteristic  life  style,  they  must  have  a  value  system  that 
controls  their  behavior.  Thus,  the  behavior  is  pervasive,  consistent, 
predictable,  and  most  importantly,  characteristic  of  the  student. 
Instructional  objectives  are  concerned  with  the  student’s  general 
patterns  of  adjustment  (personal,  social,  emotional).  Acts, 
discriminates,  displays,  influences,  listens,  modifies,  performs, 
practices,  proposes,  qualifies,  questions,  revises,  serves,  solves,  uses, 
verifies. 

Cognitive  Learning 
Theory4 

A  general  approach  that  views  learning  as  an  active  mental  process  of 

acquiring,  remembering,  and  using  knowledge.  Learning  is  evidenced  by 

a  change  in  knowledge  which  makes  a  change  in  behavior  possible. 

Learning  itself  is  not  directly  observable. 

Types  of  memory: 

•  Sensory.  The  sensory  register  is  a  system  of  receptors  which  hold 
sensory  information  for  a  very  brief  period.  Only  things  we  want  to 
remember  will  move  into  working  memory.  (Example:  Sheila  looks  at 
the  books  on  the  shelf.  For  a  brief  second  when  she  closes  her  eyes  she 
can  see  an  exact  image  of  everything  she  saw.) 

•  Working  (a.k.a.  short  term  memory).  Information  from  the  sensory 
register  focused  on  at  a  given  moment.  Information  can  be  held  for 
approximately  20  seconds  without  maintenance  rehearsal.  (Example: 
Attempt  to  remember  someone’s  name  by  saying  “his  name  is  William, 
William,  William,  William,  William.”  As  long  as  you  say  the  name,  it 
is  in  active  use  in  working  memory.  The  name  is  short  enough  to  fit  the 
limits  of  the  articulatory  loop  (1.5  sec). 

•  Long  Term.  The  permanent  store  of  information,  virtually  unlimited  in 
capacity.  Problems  with  recall  lie  in  method  of  retrieval.  (Example: 

You  remember  your  telephone  number  because  it  is  in  your  long-term 
memory.) 

Types  of  knowledge: 

•  General.  Information  that  is  useful  in  many  different  kinds  of  tasks. 
Skills  used  both  in  and  out  of  school.  (Example:  Knowing  how  to  add 
and  subtract  and  make  change.) 

•  Domain-Specific.  Information  that  generally  applies  to  only  one 
situation.  (Example:  Knowing  that  a  “roux”  is  a  butter-and-flour 
mixture  used  in  cooking.) 

•  Declarative.  Verbal  information,  lists  of  facts.  Information  that  can  be 
taught  through  lecture  or  acquired  through  books,  verbal  exchange, 
Braille,  sign  language,  etc.  (Example:  When  asked,  you  could  give 
your  address.) 

4  Cognitive  learning  theory  terms.  http://mse.bvu.edu/ipt/ipt301/iordan/Iearnterm  c.html 
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•  Procedural.  Knowledge  that  is  demonstrated  when  we  perform  a  task. 
(Example:  Knowing  what  ingredients  go  into  a  cake  (declarative 
knowledge)  but  how  to  combine  them  to  make  a  cake  is  procedural 
knowledge.) 

•  Conditional.  Knowing  when  and  why  to  use  declarative  and  procedural 
knowledge.  (Example:  You  may  know  what  a  Kokanee  is  (declarative) 
and  you  may  know  how  to  catch  one  (procedural)  but  knowing  under 
what  conditions  you  should  fish — shallow  or  deep — is  conditional 
knowledge. 

Cognitive  Learning 

(Knowledge 

Acquisition)5,6,7’8 

Demonstrated  by  knowledge  recall  and  the  intellectual  skills: 
comprehending  information,  organizing  ideas,  analyzing  and 
synthesizing  data,  applying  knowledge,  choosing  among  alternatives  in 
problem-solving,  and  evaluating  ideas  or  actions.  This  domain  is 
predominant  in  the  majority  of  courses.  The  six  levels  within  this  domain 
(with  outcome-illustrating  verbs)  are  listed  below. 

(1)  Knowledge.  The  ability  to  remember  (recall)  previously  learned 
materials.  Arranges,  defines,  describes,  duplicates,  identifies,  knows, 
labels,  lists,  matches,  memorizes,  names,  outlines,  recalls, 
recognizes,  repeats,  reproduces,  selects,  states. 

(2)  Comprehension.  The  ability  to  understand  the  meaning  of 
informational  materials.  Classifies,  comprehends,  converts,  defends, 
describes,  distinguishes,  estimates,  explains,  extends,  generalizes, 
gives  examples,  infers,  interprets,  paraphrases,  predicts,  rewrites, 
summarizes,  translates. 

(3)  Application.  The  ability  to  use  previously  learned  material  in  new 
situations.  Applies,  changes,  computes,  constructs,  demonstrates, 
discovers,  manipulates,  modifies,  operates,  predicts,  prepares, 
produces,  relates,  shows,  solves,  uses. 

(4i  Analysis.  The  ability  to  separate  material  into  its  component  parts  so 
that  its  organizational  structure  may  be  understood.  Analyzes,  breaks 
down,  compares,  contrasts,  diagrams,  deconstructs,  differentiates, 
discriminates,  distinguishes,  identifies,  illustrates,  infers,  outlines, 
points  out,  relates,  selects,  separates,  subdivides. 

(5)  Synthesis.  The  ability  to  put  parts  together  to  form  a  new  whole, 
with  an  emphasis  on  the  creation  and  formulation  of  new  meanings, 
patterns,  or  structures.  Categorizes,  combines,  compiles,  composes. 

5  Bloom* s  taxonomy.  http://www.tecweb.org/eddcvel/blooms,html 

6  Major  categories  in  the  taxonomy  of  educational  objectives. 

http://facultv.washingon.edu/-krumme/guides/bloom.html 

7  Taxonomy  of  educational  objectives,  http://www.usd.edu/admin/vpaa/assessment/taxonomies.html 

8  Bloom* s  taxonomy,  DLRN  technology  resource  guide,  Chapter  4.  http://gopher.fwl.org/tie/dlrn/blooms.html 
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creates,  designs,  devises,  explains,  generates,  modifies,  organizes, 
plans,  rearranges,  reconstructs,  relates,  reorganizes,  revises,  rewrites, 
summarizes,  tells,  writes. 

(6)  Evaluation  (highest  level).  The  ability  to  judge  the  value  of  material 
for  a  given  purpose  using  defined  criteria.  Appraises,  compares, 
concludes,  contrasts,  criticizes,  critiques,  defends,  describes, 
discriminates,  evaluates,  explains,  interprets,  justifies,  relates, 
summarizes,  supports. 

Cognitive  Task 
Analysis  (CTA) 

.  .conducted  to  ensure  that  the  software  provides  for  effective 
instruction....  Using  CTA  approaches,  it  is  possible  for  developers  to 
accurately  identify  the  knowledge  and  skills  (employed  by  experts  to 
solve  problems)  that  will  eventually  be  coded  within  the  software 
program.  Using  structured  knowledge-acquisition  processes,  CTA 
involves  interviewing  subject  matter  experts  regarding  the  knowledge 
and  skills  used  to  solve  problems  in  the  domain.  For  example,  using  the 
PARI  CTA  method,  the  following  questions  are  asked  for  each  task,  and 
related  actions  (or  simple  tasks),  performed  to  achieve  a  goal: 

Action: 

•  What  would  your  first  action  be  in  solving  this  problem? 

•  Are  there  prior  steps  you  would  have  to  take  to  perform  the  action 
(e.g.,  consult  printed  procedures?) 

Precursor. 

•  Why  are  you  taking  this  action? 

•  What  is  your  reason  in  terms  of  acquiring  information  you  need  to 
goals  you’re  attempting  to  reach? 

Result: 

•  What  does  the  result  of  obtaining  the  information  or  feedback  tell  you 
regarding  your  actions? 

Interpretation : 

•  On  the  basis  of  your  results,  what  conclusions  are  you  drawing? 

•  What  needs  to  be  done  next?”9 

Declarative 

Knowledge 

1.  Knowledge  that  is  acquired  throughout  the  learner’s  lifetime. 

2.  First  requires  the  learner  to  comprehend  the  meaning  of  incoming 
information. 

3.  Requires  the  learner  to  be  able  to  relate  new  ideas  to  old.10 

9  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Cognitive  task  analysis.  http://citl- 

s2.tamu.edu/citlsite/cta.htm 

10  Learning  strategies,  http://iivvf.edu/coehelp/club  id/lesson/strategy/acquire.htm 
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Contrasted  with  procedural,  conditional,  or  narrative  knowledge, 
declarative  knowledge  is  fundamentally  “fact-based”  knowledge. 
Declarative  knowledge  comprises  discrete  units  of  knowing,  either  very 
specific  facts  or  generalities.  For  example,  one  can  know  or  recall  a 
telephone  number  (specific)  and  one  can  know  that  telephones  are  one 
type  of  communication  device  (general).  Individual  units  of  declarative 
knowledge  can  be  linked  or  organized  into  larger  units  of  knowledge;  for 
example,  wired  telephones,  cellular  telephones,  electronic  mail,  postal 
mail,  and  radios  are  common  means  for  communicating  with  individuals. 
Declarative  knowledge  is  regarded  as  “knowing  that”  something  is  the 
case.11 

Digital 

Primarily  an  Army  classification,  digital  skills  are  those  that  support  the 
operation,  management,  and  control  of  digital  (computer)  systems  for 
battlefield  command,  control,  communication,  and  intelligence  (C3I) 
superiority.  At  the  most  basic  level,  digital  skills  are  required  for  setup, 
operation,  and  maintenance  of  the  various  equipment  systems.  At  higher 
levels,  digital  skills  are  required  for  information  processing  and 
dissemination,  situational  awareness,  decision  making,  and  other 
cognitively-mediated  requirements. 

Leadership 

There  is  no  single  definition  of  leadership  that  can  encompass  all  the 

possible  connotations. 

1.  Hanson  supplies  definitions  which  emphasize  the  role  of  people, 
processes,  and  systems  in  the  leadership  equation  to  varying 
degrees.12 

2.  Katz  &  Kahn  consider  the  essence  of  leadership  to  be  “the  influential 
increment  over  and  above  mechanical  compliance  with  the  routine 

1  o 

directives  of  the  organization.” 

3.  Lipham  writes  of  the  inherent  contradiction  in  most  definitions  of 
administrative  leadership.  “The  administrator  is  concerned  primarily 
with  maintaining,  rather  than  changing,  established  structures, 
procedures,  or  goals.  Thus,  the  administrator  may  be  viewed  as  a 
stabilizing  force. .  .We  may  define  leadership  as  the  initiation  of  a  new 
structure  for  accomplishing  [or  changing]  an  organization’s  goals  or 
objectives.”14 

4.  Getzels  argues  that  definitions  describing  the  leader  as  one  who 

11  Woolfolk,  A.  E.  (1995).  Educational  psychology ,  242.  Needham  Heights,  MA:  Allyn  &  Bacon. 

12  Hanson,  E.  M.  (1991).  Educational  administration  and  organizational  behavior ,  183.  Needham  Heights,  MA: 
Allyn  &  Bacon. 

13  Katz,  D.,  &  Kahn,  R.  (1996).  The  social  psychology  of  organizations  (2nd  ed.),  302.  New  York,  NY:  John  Wiley 
&  Sons,  Inc. 

14  Lipham,  J.  (1964).  Behavioral  science  and  educational  administration ,  122.  Chicago,  DL:  The  University  of 
Chicago  Press. 
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initiates  a  new  structure  in  the  social  systems,  as  many  definitions  do, 
are  inadequate.  He  contends,  ‘The  missing  ingredient  is  recognition 
that  leadership  depends  on  followership,  a  function  of  cooperation  or 
mutuality  with  the  leader  rather  than  forcible  domination  and  coercion 
by  the  leader.”  15 . 

5.  Getzels,  Lipham,  and  Campbell  distinguish  between  superordination, 
where  authority  is  granted  to  the  individual  by  the  institution,  and 
leadership,  where  the  authority  is  extended  by  the  followers.  The 
source  of  superordination  lies  in  vested  authority,  whereas  the  source 
of  leadership  lies  in  entrusted  authority.16 

6.  Kelly  stresses  that  it  is  the  group  that  attains  goals  and  not  the  leader, 
and  that  “leadership  is  the  performance  of  acts  which  assist  the  group 
in  achieving  certain  ends.”17 

7.  Boles  and  Davenport  say  that  leadership  is  a  process — not  a  category 
of  behavior,  a  prerogative  of  position  or  personality,  nor  a  collectivity 
of  persons.  “By  our  definition,  leadership  is  a  process  in  which  an 
individual  takes  initiative  to  assist  a  group  to  move  toward  production 
goals  that  are  acceptable,  to  maintain  the  group,  and  to  dispose  of 
these  needs  of  individuals  that  impelled  them  to  join  it.”18 

Narrative/ 

Descriptive 

A  conception  of  one  way  in  which  the  mind  organizes  knowledge.  In  the 
narrative  mode,  humans  deal  with  experience  by  constructing  stories  and 
listening  to  the  stories  of  others.  It  is  contrasted  with  paradigmatic 
thought  by  Bruner:  [paradigmatic  thought]  “deals  in  general  causes,  and 
their  establishment,  and  makes  use  of  procedures  to  assure  verifiable 
reference  and  to  test  for  empirical  truth. .  .the  narrative  mode  leads  instead 
to  good  stories,  gripping  drama,  believable  (although  not  necessarily 
‘true’)  historical  accounts. . .It  strives  to. .  .locate  the  experience  in  time 
and  place.”19 

Hands-on/ 

Perceptual-Motor 

Refers  to  the  relationship  between  human  movement  and  perceptions. 
Studies  investigating  the  link  between  movement  and  perceptual 
development  have  been  conducted  with  varying  results. 

Procedural 

Knowledge 

“Knowing  how”  to  do  something  such  as  divide  fractions  or  clean  a 
carburetor.  Notice  that  repeating  the  rule  “to  divide  fractions,  insert  the 
divisor  and  multiply”  shows  declarative  knowledge — the  student  can 

15  Getzels,  J.  (1973).  Theory  and  research  on  leadership:  Some  comments  and  alternatives.  In  Cunningham,  Luvunr, 
and  Gephart  (Eds.),  Leadership:  The  science  and  art  today ,  16.  Itasca,  IL:  F.  E.  Peacock. 

16  Getzels,  J.,  Lipham,  J.,  &  Campbell,  R.  (1968).  Educational  administration  as  a  social  process ,  135-136. 

New  York,  NY:  Harper  &  Row. 

17  Kelly,  J.  (1974).  Organizational  behavior:  An  existential-systems  approach ,  365.  Homewood,  IL: 

Richard  D.  Irwin,  Inc. 

18  Boles,  H.,  &  Davenport,  J.  (1975).  Introduction  to  educational  leadership ,  153.  New  York,  NY:  Harper  &  Row. 

19  Bruner,  J.  (1986).  Actual  minds:  Possible  worlds ,  13.  Cambridge,  MA:  Harvard  University  Press. 
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state  the  rule.  But  to  show  procedural  knowledge,  the  student  must  act. 
When  faced  with  a  fraction  to  divide,  the  student  must  divide  correctly.20 

Strategic 

Directs  knowledge,  acquisition-procedures  intentionally  employed  to 
overcome  some  deficit  in  performance  or  to  regulate  one’s  thinking  or 
performance.  A  strategy  can  be  considered  a  plan  of  action  in  which  the 
sequence  of  the  separate  cognitive  activities  is  laid  down.21 

Higher-Order 

Cognitive 

Related  to  or  involving  cognition.  Awareness,  perception,  reasoning  and 
judgment  are  all  aspects  of  cognition.22 

Cognitive  science  refers  to  the  interdisciplinary  study  of  the 
acquisition  and  use  of  knowledge.  It  includes  as  contributing  disciplines: 
artificial  intelligence,  psychology,  linguistics,  philosophy,  anthropology, 
neuroscience,  and  education.  The  cognitive  science  movement  is  far 
reaching  and  diverse,  containing  within  it  several  viewpoints. 

Cognitive  science  grew  out  of  three  developments:  the  invention  of 
computers  and  the  attempts  to  design  programs  that  could  do  the  kinds  of 
tasks  that  humans  do;  the  development  of  information  processing 
psychology  where  the  goal  was  to  specify  the  internal  processing 
involved  in  perception,  language,  memory,  and  thought;  and  the 
development  of  the  theory  of  generative  grammar  and  related  offshoots  in 
linguistics.  Cognitive  science  was  a  synthesis  concerned  with  the  kinds  of 
knowledge  that  underlie  human  cognition,  the  details  of  human  cognitive 
processing,  and  the  computational  modeling  of  those  processes. 

There  are  five  major  topic  areas  in  cognitive  science:  knowledge 
representation,  language,  learning,  thinking,  and  perception.23,24 

20  Woolfolk,  A.  E.  (1995).  Educational  psychology ,  243.  Needham  Heights,  MA:  Allyn  &  Bacon. 

21  Strategy/strategic  knowledge,  http://www.library.www.edu/cbl/rav/knowledge- 
information/strategic  knowledge.htm 

22  Dictionary.com.  http://www.dictionary.com/cgi-bin/dict.pl ?term=cognitive 

23  Cognitive  science,  http :// web  .psych .  ualberta  ,ca/~  mike/Pearl  Street/Dictionar  y/contents/C/co  gniti  ve  science,  html 

24  Eysenck,  M.  W.  (Ed.)  (1990).  The  Blackwell  dictionary  of  cognitive  psychology.  Cambridge,  MA:  Basil  Blackwell 
Ltd. 
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Analyzing  Skills  Core  thinking  skills  that  involve  clarifying  information  by  examining  parts 
and  relationships.  Identifying  attributes  and  components  (determining 
characteristics  or  the  parts  of  something),  identifying  relationships  and 
patterns  (recognizing  ways  that  elements  are  related),  identifying  main 
ideas  (identifying  the  central  element),  and  identifying  errors  (recognizing 
logical  fallacies  and  other  mistakes  and,  where  possible,  correcting  them.  5 

Audioconferencing  More  than  two  people  or  groups,  in  different  locations,  communicating  by 
voice  via  a  telecommunications  network.26 


Comprehending 


Generating  meaning  or  understanding.27 


Computer-Based  This  term  covers  the  same  types  of  programs  as  computer-learning  and 
Training  (CBT)  computer-based  learning  but  in  a  training  context  (i.e.,  computer  programs 
designed  to  interface  with  the  user  in  such  a  way  as  to  stimulate  an 
encounter  between  a  trainer  and  a  trainee).  The  term  can  refer  to  all  forms 
of  teaching  programs,  including  tutorial  dialogs  and  simulations.28 


Computer-Mediated  The  computer  equivalent  of  a  telephone  conference,  whereby  participants 
Conferencing  can  exchange  textual  messages  on  a  group  basis.  Such  exchanges  need  not 

occur  simultaneously  (synchronous),  but  can  occur  over  a  period  of  hours, 
days,  or  even  weeks,  as  time  permits  (asynchronous).29 


Core  Thinking  Skills 


30 

Cognitive  operations  used  in  thinking  processes. 


Creative  Thinking  The  ability  to  form  new  combinations  of  ideas  to  fulfill  a  need,  or  to  get 
original  or  otherwise  appropriate  results  by  the  criteria  of  the  domain  in 
question.31  Original  and  appropriate  thinking.32 


25  Thinking  skills  vocabulary  and  definitions,  http://www.adprima.com/thinkskl.htm 

26  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Glossary.  http://citl- 
s2.tamu.edu/citlsite/citl-glossarv-main.htm 

27  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 

28  Bloom’s  taxonomy,  http://www.tecweb.org/eddevel/blooms.html 

29  Ibid. 

30  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 

31  Ibid. 

32  Ibid. 
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Critical  Thinking  Reasonable,  reflective  thinking  that  is  focused  on  deciding  what  to  believe 
or  do.  Critical  thinkers  try  to  be  aware  of  their  own  biases,  to  be  objective 
and  logical.33 

Declarative  Knowledge  or  factual  information.34 

Knowledge  Stored  information  that  is  acquired  throughout  the  learner’ s  lifetime  that 

first  requires  the  learner  to  comprehend  the  meaning  of  incoming 
information  and  requires  the  learner  to  be  able  to  relate  new  ideas  to  old.35 


Effect  Size  A  standard  deviation,  or  Z  score,  with  a  range  from  about  -3.00  to  +3.00. 

A  study  with  no  difference  between  the  treatment  and  control  groups 
would  have  an  effect  size  of  0.  Any  effect  size  greater  than  0.50  is 
considered  a  major  difference,  and  a  0.20  to  0.50  effect  size  is  considered 
important.36 


Encoding  Skills 


Evaluating  (as 
applied  to 
metacognition) 

Evaluating  Skills 


Remembering  skills  that  involve  storing  information  in  long-term 
memory.37 

Assessing  one’s  current  knowledge  state 


Core  thinking  skills  that  involve  assessing  the  reasonableness  and  quality 
of  ideas.  Establishing  criteria  (setting  standards  for  making  judgments) 
and  verifying  (confirming  the  accuracy  of  claims).39 


Focusing  Skills  Core  thinking  skills  that  involve  selected  pieces  of  information  and 

ignoring  others.  Defining  problems  (clarifying  needs,  discrepancies,  or 
puzzling  situations),  and  setting  goals  (establishing  direction  and 
purpose).40 


33  Ibid. 

34  Bloom’s  taxonomy,  http://www.tecweb.org/eddevel/blooms.html 

35  Learning  strategies,  http://uwf.edu/coehelp/club  id/lesson/strategy/acauire.htm 

36  Machtmes,  K.,  &  Asher,  J.  W.  (2000).  A  meta-analysis  of  the  effectiveness  of  telecourses  in  distance  education. 
The  American  Journal  of  Distance  Education ,  14,  1. 

37  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 

38  Ibid. 

39  Ibid. 

40  Ibid. 
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Generating  Skills  Core  thinking  skills  that  involve  producing  new  information,  meaning,  or 
ideas.  Inferring  (going  beyond  available  information  to  identify  what  may 
reasonably  be  true),  predicting  (anticipating  next  events  or  the  outcome  of 
a  situation),  and  elaborating  (explaining  by  adding  details,  examples,  or 
other  relevant  information).41 


Information-  Core  thinking  skills  that  involving  bringing  to  consciousness  the  relevant 

Gathering  Skills  data  needed  for  cognitive  processing.  Observing  (obtaining  information 
through  one  or  more  senses)  and  formulating  questions  (seeing  new 
information  through  inquiry).42 


Integrating  Skills  Core  skills  that  involve  connecting  or  combining  information. 

Summarizing  (combining  information  efficiently  into  a  cohesive 
statement)  and  restructuring  (changing  existing  knowledge  structures  to 
incorporate  new  information).43 


Intelligent  Tutoring  Computer  systems  that  combine  cognitive  models  with  techniques  from 
System  artificial  intelligence  to  create  instructional  interactions  with  students.44 


Learning  Objective  The  component  actions,  knowledge,  and  skills  that  students  must  learn  if 
they  are  to  attain  mastery.  Learning  objectives  represent  the  learning 
difference  between  where  the  learner  is  now  and  where  one  wants  them  to 
be.45 


Metacognition  Knowledge  about  our  own  thinking.  Planning  how  much  time  to  allocate 

to  a  certain  task,  monitoring  how  well  we  are  doing  and  if  we  should 
change  strategies,  and  evaluating  our  efforts  to  see  if  we  have  done  an 
adequate  job.46  A  dimension  of  thinking  that  involves  control  of  self  and 
knowledge  and  control  of  process.47  Awareness  and  control  of  one’s 
thinking,  including  commitment,  attitudes,  and  attention.48 


41  Ibid. 

42  Ibid. 

43  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 

44  Bloom’s  taxonomy,  http://www.tecweb.org/eddevel/blooms.html 

45  Ibid. 

46 Cognitive  learning  theory  terms,  http://mse.bvu.edti/ipt/ipt301/iordan/leamterm  c.html 

47  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 

48  Ibid. 
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Organizing  Skills  Core  thinking  skills  that  involve  arranging  information  so  that  it  can  be 
used  more  effectively.  Comparing  (noting  similarities  and  differences 
between  or  among  entities),  classifying  (grouping  and  labeling  entities  on 
the  basis  of  their  attributes),  ordering  (sequencing  entities  according  to  a 
given  criterion),  and  representing  (changing  the  form,  but  not  the 
substance  of  information).49 

Problem  Solving  Analyzing  a  perplexing  or  difficult  situation  for  the  purpose  of  generating 
a  solution.50 

Procedural  Knowledge  of  how  to  perform  various  tasks.51 

Knowledge 

Recalling  Skills  Remembering  skills  that  involve  retrieving  information  from  long-term 
memory.52 

Remembering  Skills  Core  thinking  skills  that  involve  conscious  efforts  to  store  and  retrieve 
information.  Encoding  (storing  information  in  long-term  memory)  and 
recalling  (retrieving  information  from  long-term  memory).53 

Rote  Memorization  Remembering  information  by  repetition  without  necessarily  understanding 
the  meaning  of  the  information.  Considered  to  be  a  very  low  level  of 
learning  but  occasionally  necessary.  Memorization  can  be  hampered  by 
the  serial  position  effect  (the  tendency  to  remember  the  beginning  and  the 
end  but  not  the  middle  of  the  list).54 

Teleconferencing  Two-way  electronic  communication  between  two  or  more  groups,  or  three 
or  more  individuals,  who  are  in  separate  locations.  Includes  group 
communication  via  audio,  audiographs,  video,  and  computer  systems.55 

Thinking  The  process  of  creating  a  structured  series  of  connective  transactions 

between  items  of  perceived  information.56 


51  Bloom’s  taxonomy,  http://www.tecweb.org/eddevel/blooms.html 

52  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 


54  Dictionary.com.  http://www.dictionarv.com/cgi-bin/diet.pl?term=memorization 

55  Bloom’s  taxonomy,  http://www.tecweb.org/eddevel/blooms.html 

56  Copyright  2000  ©  CITL,  Cognition  &  Instructional  Technologies  Laboratories,  Intelligent  tutors:  Computational 
intelligence,  http://citl-s2.tamu.edu/citlsite/intelligent-tutors.htm 
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ACRONYM  LIST 


ADL 

Advanced  Distributed  Learning 

AEC 

Automated  Electronic  Classroom 

AFDLO 

Air  Force  Distance  Learning  Office 

AFIT 

Air  Force  Institute  of  Technology 

ARI 

Army  Research  Institute 

ASTD 

American  Society  of  Training  and  Development 

ATN 

Air  Technology  Network 

C3I 

Command,  Control,  Communication,  and  Intelligence 

CMC 

Computer-mediated  Communication 

CTA 

Cognitive  Task  Analysis 

DL 

Distributed  Learning 

DO 

Delivery  Order 

DSTR 

Digital  Skills  Training  Research 

ES 

Effect  Size 

FA 

Field  Artillery 

GETN 

Government  Education  and  Training  Network 

ISD 

Instructional  Systems  Development 

KOs 

Knowledge  Objects 

KSA 

Knowledge,  Skills,  and  Attitudes 

MI 

Military  Intelligence 

NPS 

Naval  Postgraduate  School 

RCOs 

Reusable  Content  Objects 

RLOs 

Reusable  Learning  Objects 

SCORM 

Sharable  Courseware  Object  Reference  Model 

SD 

Standard  Deviation 

TRAD AM 

Training  Dekuvery  Assessment  Model 

VTT 

Video  Teletraining 
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